Migration applied to one tenant only

Migration applied to one tenant only
Type: incident simulation · Topic: Migrations & Releases · Level: L3 · Duration: 10–15 min
Launch: ride postgres start stage-09/08-migration-applied-to-one-tenant-only

POSTMORTEM (root cause · how it was found · the fix · lesson)
Root cause: a multi-tenant rollout was supposed to add `orders.release_id` to every
tenant database, but it only completed for tenant_a. tenant_b and tenant_c stayed on
the old schema, so the release was half-applied — green for one tenant, broken for
the rest.

How it was found: comparing information_schema.columns for orders across tenant_a,
tenant_b and tenant_c showed release_id present only in tenant_a.

The fix: apply the migration to the tenants that missed it:
  \connect tenant_b
  ALTER TABLE orders ADD COLUMN release_id text;
  \connect tenant_c
  ALTER TABLE orders ADD COLUMN release_id text;

Lesson: a tenant rollout is not complete because one tenant migrated. A release is
only successful once every intended database/tenant is verified — automate the
fan-out and check each target. Validating the one tenant that already migrated, or
reaching for an index, hides the gap instead of closing it.

INVESTIGATION HINTS (the staged path to diagnose and fix)
1. The release migration (ALTER TABLE orders ADD COLUMN release_id) reported success, but only one tenant actually got it. Check EVERY tenant database, not just the one you happened to look at: \connect tenant_a / tenant_b / tenant_c and inspect information_schema.columns for orders.
2. tenant_a migrated; tenant_b and tenant_c are still on the old schema (no release_id). A rollout isn't done because one tenant succeeded — every intended tenant must be verified.
3. Apply the migration to the tenants that missed it: \connect tenant_b; ALTER TABLE orders ADD COLUMN release_id text; then the same for tenant_c. Validating tenant_a again — or adding an index — proves nothing.