The Test Suite That Was Quietly Live

A test suite is the one place in a codebase where you're allowed to be reckless. Create rows, delete rows, call the destructive function, run it a hundred times — that's the whole point of testing in a sandbox instead of the real thing. Nobody sits down and decides to point their tests at production. It's the kind of mistake that happens by default, quietly, and then just never gets questioned because it seems to work.

19 test files for the CLI ran against whatever database was configured in that app's own environment file. And what was configured there wasn't a sandbox — it was the same shared production Postgres database every real user's chats, scheduled jobs, memories, preferences, and permission rules live in.

These weren't read-only tests either. They hard-coded a real user row and created and deleted real chats, real scheduled jobs, real memories against it — some of them calling a bulk-delete across several of those tables directly.

Nothing about running the test suite was supposed to be able to touch a real person's data. It could.

Not by noticing. A routine credential rotation, done for unrelated reasons, changed the database password — and every one of those 19 test files immediately failed with an authentication error. That failure is what surfaced it. Before that, the tests had been passing the whole time, which tells you nothing about whether they were safe — only that nothing had gone wrong yet.

We looked for evidence of real damage from this and didn't find any. That's not the same as a guarantee, and we're not going to pretend it is — it's just the honest state of what we know. The exposure itself was real for as long as it had been set up this way, whether or not anything ever actually broke because of it.

Tests now default to a disposable database, created fresh and thrown away on every run, with the real schema pushed onto it and exactly the one seed row the tests assume. Running anything against a real Postgres database at all now requires deliberately opting in with an explicit environment flag — and even then, a host allow-list refuses to connect to anything that looks like the real production address.

Fixing this properly also meant finding what the drift had been quietly hiding: a real timeout bug in an approval prompt that could freeze indefinitely, and a test that had been asserting the wrong behavior for a bug we'd already fixed elsewhere, because the suite hadn't been able to actually run cleanly enough to catch it.

"Be more careful" isn't a fix — it's a hope, and hope isn't a safeguard. The actual fix is a test suite that's structurally incapable of reaching real data unless someone deliberately flips a switch to let it. That's the difference between a mistake that could happen again and one that can't.