Backward-incompatible migration

Backward-incompatible migration
Type: incident simulation · Topic: Migrations & Releases · Level: L4 · Duration: 10–15 min
Launch: ride postgres start stage-09/05-backward-incompatible-migration

POSTMORTEM (root cause · how it was found · the fix · lesson)
Root cause: a release migration dropped `app_db.orders.legacy_status`, but the
deploy was still rolling — the old app version (which reads legacy_status) was
running alongside the new one. The schema change was backward-incompatible, so
every old-app request that touched legacy_status broke.

How it was found: information_schema.columns showed legacy_status missing from
orders, and pg_stat_activity showed the old app version still connected
(application_name LIKE 'old_app%').

The fix (restore compatibility): re-add the column and backfill it, without
breaking the new app:
  ALTER TABLE orders ADD COLUMN legacy_status text;
  UPDATE orders SET legacy_status = status WHERE legacy_status IS NULL;

Lesson: during rolling deploys, schema changes must be backward-compatible. Expand
first (add the new shape), deploy the app compatibility layer, backfill, switch
reads/writes, and only contract (drop/rename) after every old app version is gone.
Killing the old app sessions or adding an index fixes nothing.

INVESTIGATION HINTS (the staged path to diagnose and fix)
1. A migration changed app_db.orders during a rolling deploy and the old app version (application_name LIKE 'old_app%', still connected) started failing. Compare the columns it expects against what's there now.
2. List orders' columns via information_schema.columns — the legacy_status column the old app reads is gone. Dropping/renaming a column mid-deploy is backward-incompatible: old and new app versions run at the same time.
3. Restore compatibility in app_db without breaking the new app: re-add legacy_status and backfill it from status (ALTER TABLE orders ADD COLUMN legacy_status text; UPDATE orders SET legacy_status = status). Don't just kill the old app, and don't add an index.