Tablespace full

Tablespace full
Type: incident simulation · Topic: Storage & Backup · Level: L2 · Duration: 10–15 min
Launch: ride postgres start stage-05/07-tablespace-full

POSTMORTEM (root cause · how it was found · the fix · lesson)
Root cause: a single tablespace (fastspace, on its own bounded mount) was nearly
filled by a junk table, so writes to objects in that tablespace failed with 'no
space left' — while the rest of the cluster, on other storage, was unaffected.
The failure was localized to one tablespace, not the whole database.

How it was found: pg_tablespace_size showed fastspace near its limit; pg_tables
showed the junk table living there.

The mitigation: drop the table filling the tablespace; usage dropped and writes
to it succeeded again.

Lesson: read storage pressure at the right granularity — a full tablespace is
not a whole-cluster outage. Find what's consuming it (pg_tablespace_size /
pg_tables), and relieve that tablespace specifically. Don't reach for indexes or
max_connections.

INVESTIGATION HINTS (the staged path to diagnose and fix)
1. Writes to one tablespace fail with 'no space', but the rest of the database is fine — the pressure is localized to a tablespace, not the whole cluster. List them: SELECT spcname, pg_size_pretty(pg_tablespace_size(oid)) FROM pg_tablespace;
2. One tablespace (fastspace) is nearly full. Find what's filling it: a junk table lives there. SELECT tablename, tablespace FROM pg_tables WHERE tablespace = 'fastspace';
3. Reclaim the space by dropping the junk table that fills it: DROP TABLE tbl_fill; the tablespace usage drops. Don't add indexes or raise max_connections — this is localized storage, not a global outage.