pg_hba blocks the application

pg_hba blocks the application
Type: incident simulation · Topic: Security & Access · Level: L2 · Duration: 10–15 min
Launch: ride postgres start stage-10/01-pg-hba-blocks-application

POSTMORTEM (root cause · how it was found · the fix · lesson)
Root cause: a `pg_hba.conf` rule rejected `app_user`'s connection to `app_db`. The
database was healthy and the admin (`pgpg`) superuser connected fine, so it looked
like a partial outage — but it was an access-control problem. pg_hba is evaluated
top-down, first match wins, and a `reject` rule sat ahead of the normal `trust`/auth
line for that user+database.

How it was found: the admin connection worked while the app connection failed;
pg_settings located the `hba_file`, and the offending reject line was the cause.

The fix: remove the reject rule from pg_hba.conf and reload —
`pgpg action restore-app-pg-hba` (it edits only the container-internal
`$PGDATA/pg_hba.conf` and runs `pg_reload_conf()`).

Lesson: distinguish "database down" from "access denied". A failing app connection
while admin works points at pg_hba / authentication, not capacity. Don't reach for
max_connections, and never "fix" access by granting superuser or widening auth for
everyone — a pg_hba reject blocks superusers too, so that wouldn't even work here.

INVESTIGATION HINTS (the staged path to diagnose and fix)
1. The database is up — your admin (pgpg) connection works — but the application role can't connect. That's an access/authentication problem, not an outage. Don't treat Postgres as down.
2. Check where authentication is decided: SELECT name, setting FROM pg_settings WHERE name IN ('hba_file','config_file'); A pg_hba.conf rule is rejecting app_user's connection to app_db. pg_hba is matched top-down, first match wins.
3. Fix it safely: remove the reject rule and reload — run `pgpg action restore-app-pg-hba`. Don't raise max_connections, don't grant app_user superuser (a reject rule blocks superusers too), don't widen authentication for everyone.