Five Rules That Cut My Homelab Logs by 95%

It is Tuesday morning. I am cleaning up disk space because my homelab server is screaming at 94% capacity. I run du -sh /var/log/* and the output stops me: 200GB. Just logs. Uncompressed. From 18 services I forgot I was running.

The largest offender: docker/json-file logs from a Prometheus container that has been logging every single scrape target every 15 seconds for eight months. 87GB by itself. Next: nginx access logs, 34GB, no rotation configured. Then Grafana, 22GB, debug level still on from a troubleshooting session in November.

I deleted 190GB in about ten minutes. Not because I was reckless. Because none of it was indexed. None of it was searchable. None of it had ever been read.

This is not a post about building a centralized logging stack with Elasticsearch and Kibana. That is a weekend project that becomes a second job. This is a post about the five rules I use now to keep logs under control on a single-server homelab. The rules that keep me at 10GB total instead of 200GB.

The Log Hoarding Problem

Here is what happened: I set up each service with default logging. Docker’s default is “log everything forever until the disk is full.” Systemd’s default is “keep everything in the journal.” Nginx’s default is “write access logs with no rotation.”

Over eight months, this accumulated. I did not notice because the server has a 1TB drive. I only noticed when I hit 94% capacity and started getting alerts.

The uncomfortable truth: I had never searched those logs. Not once. When I needed to debug something, I would docker logs --tail 200 the specific container. I would grep the last hour of syslog. I would check the Grafana dashboard. I never once needed the 87GB of Prometheus scrape logs from March.

I was hoarding logs the way I hoard browser tabs: “I might need this later.” I never did.

Rule 1: Rotate Everything, Compress Always

The first fix is logrotate. Every service that writes to a file gets a rotation config. The pattern I use:

/var/log/nginx/*.log {
    daily
    missingok
    rotate 7
    compress
    delaycompress
    notifempty
    create 0640 www-data adm
    sharedscripts
    postrotate
        [ -f /var/run/nginx.pid ] && kill -USR1 $(cat /var/run/nginx.pid)
    endscript
}

This keeps 7 days of logs, compresses them after one day, and creates fresh files with proper permissions. The delaycompress is important: it keeps the most recent rotated file uncompressed so you can still grep it quickly without decompressing.

For Docker containers, I set logging options in the daemon.json:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

This caps each container at 30MB total (3 files × 10MB). If a container is logging more than that, you have a problem to fix, not a log file to keep.

Rule 2: Debug Level Is Temporary

Here is what happened with Grafana: in November, I was troubleshooting a datasource issue. I set the log level to DEBUG to see what was going on. I fixed the issue. I forgot to change it back.

Eight months later, Grafana had written 22GB of debug logs. Every panel refresh, every query, every user action, all logged at maximum verbosity.

The fix: I made a rule that debug logging requires a calendar reminder. When I enable debug level on anything, I immediately set a reminder for 24 hours later to turn it off. If the issue is not fixed in 24 hours, I either need to escalate or I need to stop debugging and think.

The systemd journal gets the same treatment:

# /etc/systemd/journald.conf
[Journal]
SystemMaxUse=500M
SystemKeepFree=2G
SystemMaxFileSize=50M
SystemMaxFiles=10

This caps the entire journal at 500MB. If something needs more logging than that, it needs its own dedicated log file with rotation.

Rule 3: Metrics Beat Logs for Most Things

I realized something uncomfortable: most of the logs I was hoarding were actually metrics in disguise.

Prometheus scraping every 15 seconds? That is not a log problem. That is a metrics problem. The scrape targets should be in Grafana, not in a text file.

Nginx access logs? I already have a Grafana dashboard showing requests per second, error rates, and response times. I do not need the raw logs unless I am debugging a specific request.

The rule now: if I want to know “how often” or “how many” or “how slow,” that is a metric. It goes to Prometheus or a simple counter. If I want to know “what exactly happened,” that is a log. It gets written at INFO level with rotation.

This cut my logging volume by about 60%. The Prometheus container alone went from 87GB to zero, because I stopped logging every scrape and just let Prometheus do what Prometheus does.

Rule 4: Centralize or Delete

Here is the hard truth about homelab logging: if you are not going to search it, do not keep it.

I am not running an ELK stack. I am not running Loki. I am running one server with 18 services, and I am the only user. The probability that I need to search logs from three weeks ago is approximately zero.

So the rule is: logs stay for 7 days, compressed, on the local disk. If I need to search them, I grep them within that week. After 7 days, they are gone.

The only exception: security-relevant logs. Auth failures, sudo usage, SSH logins. Those go to a separate rotation with 30-day retention:

/var/log/auth.log {
    weekly
    rotate 4
    compress
    delaycompress
    missingok
    notifempty
    create 0640 root adm
}

Four weeks of auth logs. If I have not noticed a brute force attempt in four weeks, I am not going to notice it in five.

Rule 5: The Log Budget

This is the rule that actually keeps me honest: I have a log budget.

The /var/log directory gets 10GB. That is it. If it exceeds 10GB, something is wrong and I need to fix it, not expand the budget.

I set up a simple cron job that checks daily:

#!/bin/bash
# /usr/local/bin/check-log-budget.sh
LOG_SIZE=$(du -sm /var/log | cut -f1)
if [ "$LOG_SIZE" -gt 10000 ]; then
    echo "WARNING: /var/log is ${LOG_SIZE}MB (budget: 10GB)" | \
        mail -s "Log Budget Exceeded" admin@localhost
fi

This has fired twice in three months. Both times, it was because a new service shipped with default logging and no rotation. I fixed the config, deleted the excess, and moved on.

The budget forces discipline. When you have infinite disk, you hoard. When you have 10GB, you make choices.

The Config Changes: What I Actually Modified

Here is the actual work I did to get from 200GB to 10GB:

1. Docker daemon logging limits

// /etc/docker/daemon.json
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

Then restart Docker: systemctl restart docker

2. Nginx logrotate

# /etc/logrotate.d/nginx-custom
/var/log/nginx/*.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    create 0640 www-data adm
    sharedscripts
    postrotate
        [ -f /var/run/nginx.pid ] && kill -USR1 $(cat /var/run/nginx.pid)
    endscript
}

3. Grafana log level

# /etc/grafana/grafana.ini
[log]
mode = console file
level = info
filters =

Changed from level = debug to level = info.

4. Prometheus scrape logging disabled

# /etc/prometheus/prometheus.yml
global:
  scrape_interval: 15s
  # Removed: scrape_log_level: debug

5. Systemd journal limits

# /etc/systemd/journald.conf
[Journal]
SystemMaxUse=500M
SystemKeepFree=2G

Then restart: systemctl restart systemd-journald

6. Log budget monitoring

# /etc/cron.daily/check-log-budget
#!/bin/bash
LOG_SIZE=$(du -sm /var/log | cut -f1)
if [ "$LOG_SIZE" -gt 10000 ]; then
    echo "WARNING: /var/log is ${LOG_SIZE}MB" | \
        mail -s "Log Budget Exceeded" admin@localhost
fi

Total time: about 90 minutes. Total disk recovered: 190GB.

The Real Lesson

The real lesson is not about logrotate configs. It is about default discipline.

Every service I install now gets a logging config before it starts. Docker logging limits. Logrotate rules. Log level set to INFO, not DEBUG. If the service does not support any of these, I think twice about running it.

I also ask: “What log would I actually search if something broke?” If the answer is “none, I would just check the Grafana dashboard,” then the log does not need to exist.

My homelab is at 10GB of logs now. It has been stable at 10GB for three months. The log budget alert has fired twice, both times for new services that needed config fixes.

I have not needed to search logs older than 7 days once in those three months.

Sometimes the answer is not “build a better logging stack.” The answer is “delete most of it.”

Related reading: how I cut AI agent costs by 60% with quantized models in a homelab and the self-hosted monitoring stack I actually use.