Backup target full

Backup target full
Type: incident simulation · Topic: Storage & Backup · Level: L2 · Duration: 10–15 min
Launch: ride postgres start stage-05/04-backup-target-full

POSTMORTEM (root cause · how it was found · the fix · lesson)
Root cause: the base-backup destination was almost full, so pg_basebackup failed
for lack of space. The database itself was perfectly healthy — only backup safety
was compromised, which is easy to miss because queries keep working.

How it was found: the backup job failed; inspecting the destination
(pg_ls_dir on the bounded backup dir) showed it nearly full of old data/junk.

The mitigation: free/rotate the backup destination, then retry the backup
(`pgpg action cleanup-backup-target` + `run-backup`); a complete base backup then
existed.

Lesson: a failing backup is an incident even when the DB is fine — monitor it.
Fix the destination (rotate/expand/offload) and re-run + verify the backup. Don't
treat it as a query problem (indexes, VACUUM) and don't delete backup files
blindly without re-taking and verifying a good backup.

INVESTIGATION HINTS (the staged path to diagnose and fix)
1. The database is healthy and queries work, but the nightly base backup is failing — backup safety is broken even though nothing is wrong with the DB. This is a storage-destination problem, not a query/index one.
2. Inspect the backup destination (you're superuser): SELECT * FROM pg_ls_dir('/tmp/pgpg_backup'); it's nearly full of junk, so pg_basebackup has no room.
3. Free the destination and retry the backup: `pgpg action <session> cleanup-backup-target` then `pgpg action <session> run-backup`. A complete base backup then exists. Don't add indexes or VACUUM — neither has anything to do with the backup target.