🧯 Disk Full Panic? One Habit Kills It

General
1

TL;DR: Before you add another TB of storage, check what’s been writing logs for 90 days. A monthly 10-minute “disk diet” stops the 2am panic before it starts.

The Hook

It’s 2:47am. Your agent’s vector store stopped responding. You SSH in, heart pounding, and see it: No space left on device. The invoice from last week’s client is stuck in a queue because a single Docker container wrote 40GB of debug logs into a volume you forgot existed.

We’ve all been there. The fix isn’t more storage—it’s a habit.

The Pattern

I used to treat disk space like weather: check it only when it stormed. Then I realized that on small boxes running multiple agents, the disk doesn’t fill up gradually. It fills up in a Tuesday afternoon spike, usually from a model retry loop or a misconfigured cron job.

The habit that fixed it is boring: a monthly purge date.

Pick a recurring calendar slot—say, the first Friday at 3pm. Set a timer for 10 minutes. Do three things, in this order:

  1. Find the big three. Run du -h --max-depth=1 /var | sort -h and look for anything over 1GB that isn’t a database you know about.
  2. Check log rotation. If you don’t have logrotate configured for every container, that’s your problem. I once found a Python script writing to stdout with no rotation—it ate 12GB in three weeks.
  3. Delete the obvious. Old model caches, stale Python wheels, and any file modified over 90 days ago in /tmp. You won’t miss them.

The first time, you’ll find a 30GB surprise. The second time, it’s a 2GB cleanup. By month three, it’s a 30-second glance.

The Takeaway

  • Set a monthly calendar reminder titled “Disk Diet” — treat it like paying rent.
  • Default to logrotate with maxsize 100M on every new container you spin up. Do it now, not during the incident.
  • Don’t buy more storage to fix a hygiene problem. It’s like buying a bigger fridge to ignore the mold.

AI models and agents are getting smarter every week—even the controversial ones—but they still write logs like it’s 1999. You can’t control what they generate, but you can control what you keep.

We run these systems daily on our own boxes at hub.sqs.chat, and this habit is the only reason we sleep through the night. Your future 2am self will thank you.

by Jonas Reyes — the builder’s desk, Side Quest Studios
AI-assisted, curated for Side Quest Studios.

References

  1. even the controversial ones — OpenAI launches Astra, its powerful (and controversial) new model | TechCrunch