Log file growth

Log file growth
Type: incident simulation · Topic: Storage & Backup · Level: L2 · Duration: 10–15 min
Launch: ride postgres start stage-05/08-log-file-growth

POSTMORTEM (root cause · how it was found · the fix · lesson)
Root cause: a noisy workload (log_spammer) flooded the server log with verbose
entries, so the collected log directory filled fast. The database itself was
healthy — queries worked, pg_wal was normal — but the log volume threatened the
disk. It's easy to misread as a table/WAL/index problem.

How it was found: pg_ls_logdir() showed the log directory growing continuously;
pg_stat_activity showed one application flooding the log; the logging settings
(logging_collector/log_statement) confirmed verbose logging into a bounded dir.

The mitigation: stop the runaway log-spamming workload; the log directory then
stopped growing.

Lesson: logs are storage too — trace log-volume pressure to the workload (or the
verbose setting) producing it and stop/fix that. Don't disable logging entirely
as a reflex (you lose your audit trail), don't add indexes (unrelated), and never
delete log files by hand — fix the producer and let rotation reclaim space.

INVESTIGATION HINTS (the staged path to diagnose and fix)
1. The database responds fine and pg_wal is normal, but a volume is filling. Look where logs go: SELECT name, setting FROM pg_settings WHERE name IN ('logging_collector','log_directory','log_statement'); then size them: SELECT name, size FROM pg_ls_logdir() ORDER BY size DESC; the log dir keeps growing.
2. It's log volume, not tables/WAL/indexes. Find the culprit: SELECT application_name, state, count(*) FROM pg_stat_activity GROUP BY 1,2 ORDER BY 3 DESC; one app (log_spammer) is flooding the log.
3. Stop the noisy workload: SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE application_name LIKE 'log_spammer%'; the log dir then stops growing. Don't disable logging wholesale, don't add indexes, and never delete log files by hand.