Scenario · Multi-Database
Backup skipped one database
A sandboxed PostgreSQL incident — investigate with your own tools, submit a fix, and get deterministic Detect / Fix / Trap scoring.
L2 · 10–15 min · runs locally in Docker
Launch
Start this scenario
Boot it in a real PostgreSQL sandbox and investigate with psql, EXPLAIN and pg_stat_statements.
ride postgres start stage-06/03-backup-skipped-one-databasePart of these paths
Show the postmortem & investigation hints spoilers
Backup skipped one database
Type: incident simulation · Topic: Multi-Database · Level: L2 · Duration: 10–15 min
Launch: ride postgres start stage-06/03-backup-skipped-one-database
POSTMORTEM (root cause · how it was found · the fix · lesson)
Root cause: the backup job dumped app_db but silently skipped billing_db, so the
cluster's backup was incomplete. Because app_db's dump existed and the job
reported success, the gap went unnoticed — until a restore would have needed
billing_db and found nothing.
How it was found: listing the databases that must be protected and comparing
against the backup destination showed billing_db.sql missing.
The mitigation: take a complete multi-database backup covering every database
(`pgpg action take-complete-multidb-backup`), then verify each dump is present.
Lesson: in a multi-database cluster, "a backup exists" is not "every database is
backed up." Enumerate the databases and validate coverage. Checking only the first
database, or trusting the job's exit code, hides the gap. An index is irrelevant.
INVESTIGATION HINTS (the staged path to diagnose and fix)
1. The nightly backup 'succeeded', but does it cover every database? List the databases you must protect: SELECT datname FROM pg_database WHERE datname NOT IN ('postgres','template0','template1','incident'); then see what's actually in the destination: SELECT * FROM pg_ls_dir('/tmp/pgpg_backup');
2. Only app_db.sql is there — billing_db was skipped. A backup that exists for one database is not a backup of the cluster.
3. Take a complete multi-database backup: `pgpg action <session> take-complete-multidb-backup`, then re-check that both app_db.sql and billing_db.sql exist. Don't add indexes, and don't trust the job just because app_db is present.