Release 3848f7a0e: API refusals and model-list errors

A new database lookup on GET /v1/models overloaded the production database. That slowed the job that writes request records, and because gateway capture waited on those records, the gateway refused API requests for 27 minutes. A database hotfix ended it at 19:52 UTC.

What went wrong

MR !1124 made GET /v1/models list each customer's free :free model names. To do that it called a database function on every listing. Clients call this endpoint about 8 times a second, and on production data each call took about 1.4 seconds of database CPU. Traffic did not change; the cost per call did.

1. 1
   Every model listing ran an expensive query.

   18,265 calls in under an hour, averaging 3.3 s under load: about 60,000 s of database time.

2. 2
   The database ran out of CPU.

   Up to 56 copies of the query ran at once. Everything else on the database slowed down.

3. 3
   The request-record writer fell behind.

   It went from about 28,000 to 15,000 receipts per 5 minutes, so request rows landed late.

4. 4
   Capture waited for those rows, and its buffer filled.

   Each gateway worker's pending captures went from about 1 to 400 to 760 (about 500 MB).

5. 5
   The gateway refused new requests it had to capture.

   503 capture_unavailable, mostly on uncensored models but on every model.

Two other things made it worse. The release reset the control plane to 4 pods, and a missing setting (BALANCE_LEDGER_WRITEBACK_REPLICAS) let one pod hold four of the eight record streams, so one customer's records stayed about 20 minutes late after the database recovered. None of the other MRs in the release caused the incident; !1125 has a separate bug, fixed by !1142.

Timeline (UTC)

Why testing missed it

• The tests never touched a real database. !1124's unit tests used a fake cursor and only checked that the query was sent.
• CI and staging are too small to show it. The cost comes from production's catalog (1,204 models, 18 promotions) times production's call rate (about 1,400 callers). Neither exists outside production.
• A quick check looks fine. Run by hand with real values the query takes 20 ms; only inside the function does Postgres pick the slow plan.
• Nobody asked for a load estimate. The function was already known to cost 0.4 s per call; moving it onto a path called 8 times a second needed that question asked in review.
• The release was large. About 14 MRs and three direct commits shipped together, which slowed diagnosis.

How we fixed it

• Live
  Database hotfix. The lookup returns nothing for the gateway's model listing, so /v1/models works again without the :free names (its behavior before !1124). Everything else that uses the function is unchanged.
• Live
  Control plane scaled to 24 pods, and the writer setting fixed so each pod takes at most 2 record streams.
• Done
  Crons: the geo-country cron removed; the other two paused crons resumed.
• Failed
  Gateway rollback. Porter said no rollback was needed, the old build was not allowed to start outside a release, and the stuck release record blocked recovery. The gateway stayed on the new build.
• Merged
  Recovery for a stuck release (!1151). Recovery now works after main has moved on, and accepts the two apps changed by hand during the incident. Kion is running the recovery and the next release.
• Merged
  Writer setting kept through releases (!1155, Kion).
• Waiting
  Revert !1124 (!1141) and fix !1125 (!1142). Both are on main and ship in the next release, once the stuck release record is cleared.

What we are changing

• Ready
  Fast rollback (!1145). Each release starts a second gateway fleet and moves traffic over gradually (our own orgs first, then 1%, 10%, 50%, with protected customers such as Koodos and Screenpipe moved last, at 100%), halting by itself if errors rise. For the first hour, rollback is a traffic switch back to the old fleet, whatever state the release is in; a preview proved the switch back in 6.3 seconds. After merge it needs six activation steps, ending with a staging rehearsal.
• Merged
  Capture never blocks the API (!1147). A full or failed capture drops the capture, counts it and alerts; the request still succeeds, uncensored models included. A local-disk buffer absorbs short slowdowns. The engine side is published (0.7.238). It was fault- and load-tested in a preview environment before merge, and ships in the next release.
• Merged
  Accounting isolated from the dashboard (!1146). The record writer runs in its own fixed 4-pod service with a health check, so control-plane scaling cannot starve it.
• Merged
  Releases stop resetting scaling (!1144): the control plane keeps a floor of 24 pods.
• Ready
  The :free listing done properly (!1149): built in worker memory every 5 minutes, with no database call per request.
• Merged
  Crons off the database (!1154, !1150). An audit of all 150 jobs found they use about 36 hours of database time a day. The fixes cut about 70,000 seconds a day and stop the failing promotion job. The geo-country job is gone; new orgs get their country once at signup.
• Planned
  Load-tested hot paths. Any MR that adds a database read to a per-request path states the expected calls per second and cost per call, and is tested against production-sized data.
• Planned
  Alerts on capture refusals, /v1/models errors, database CPU and slow new queries. We learned about this incident from other people.
• Rule
  Whoever starts a production deploy stays and watches it to the end.
• Rule
  Smaller releases, real merge holds. Holds must reach every session, and nothing goes to main without an MR and CI.

How rollback will work (!1145)

Today a gateway rollback cannot work: the old build is refused when it starts outside a release, and Porter reports there is nothing to roll back. !1145 replaces that with a traffic switch between two gateway fleets, called colors.

1. Two fleets. The gateway runs as two copies, color a and color b. One serves all traffic (the stable color); the other is idle with zero pods.
2. A release starts the idle color on the new build, already scaled to the serving color's size, while the old build keeps serving.
3. Traffic moves gradually. The edge reads a routing record every 2 seconds and splits traffic by API key, so each customer sticks to one color. Our own orgs and the smoke-test key go first (the cohort), then 1%, 10% and 50% of everyone else, five minutes per step. A protected list of key customers stays on the old version until the final 100% step, after the rest of the release has proved itself (being added in a follow-up MR).
4. It stops by itself. At every step the release checks the new color's 5xx share, p95 latency, capture drops, writeback lag, database connections and database CPU. If any crosses its threshold, the release halts and traffic returns to the old color. The thresholds are configuration, not code.
5. An hour of safety. After 100% the release bakes for an hour with the old color still running. During the steps and the bake, one click (Admin, then Platform, then Production releases, typing ROLL BACK, or one CI job) sends 100% of traffic back to the old color within seconds, whatever state the release is in. A preview measured the switch back at 6.3 seconds.
6. Commit. After a clean bake the old color is scaled to zero (its settings kept) and becomes the idle color for the next release.
7. After the bake. Rollback is then a re-release of the previous build onto the idle color with the same gradual steps: minutes, not seconds. The control plane, edge and web are restored from that release's saved configuration.

What it does not undo: database migrations. They apply before any pod moves and are never rolled back, which is why every migration must stay readable by the previous build (the expand-then-contract rule, enforced in CI). A rollback switches code, not schema.

Cost: during a release the gateway fleet is doubled for about an hour and a half.

Can we roll back the next release?

Not with the fast switch. !1145 ships turned off: the release that carries it still runs one gateway fleet, so there is no second color to switch to. Its rollback request does work, but without colors it waits for the old build's fleet to come back, which takes minutes, and the old build here is several days old. For the next release the protections are:

• keep it small, and have whoever starts it watch it to the end;
• the kill switches we used today: database hotfixes, pausing crons, scaling the control plane, and the capture kill switch;
• fix forward, which is fast now that CI is green and the release machinery is unjammed.

Fast rollback is available from the first release after colors are turned on: run the index, rehearse on staging, then turn colors on in production.

How isolated is the API from the dashboard?

Serving /v1 never depended on the dashboard: gateway workers go straight to the database, which is why the API stayed up when the dashboard was slow. After the next release and !1147:

• Isolated
  Accounting vs. dashboard scaling. The record writer runs in its own fixed service, so dashboard pod counts cannot starve it.
• Isolated
  Capture vs. API traffic (with !1147). A capture problem drops the capture and alerts; it never refuses a request.
• Shared
  One Postgres database. This was the root of the incident: a query that overloads the database still slows the API's accounting along with the dashboard. Closing this means read replicas for dashboard and analytics reads, more analytics on ClickHouse, and keeping per-request database reads off hot paths.
• Partly
  Deploy unit and edge. The record writer still lives in the dashboard's Porter app (separate pods, shared config), and one edge app serves both /v1 and the dashboard API relay.

How to isolate them completely

The principle: a request to /v1 should never wait on anything the dashboard can slow down. Today the database is the shared dependency, so the work is to take the API's request path off it and take the dashboard's heavy reads off the primary. In order of value:

Recommendation: do steps 1 to 4. Together they give practical isolation (a slow dashboard or a heavy report can no longer affect API requests) at the cost of one read replica and a few weeks of work. Step 5 is only worth it if the API still feels dashboard load after that, which is unlikely once the request path no longer waits on Postgres.

Release order

Each release is watched to the end by whoever starts it. Nothing ships until the stuck release record from this incident is cleared, which also unblocks catalog publishing (blocked since 18:24 UTC because the gateway runs a release that was never promoted).

1. Next release (main): the !1124 revert (!1141), the breaker fix (!1142), the scaling floor (!1144), accounting isolation (!1146), capture never blocking the API (!1147), the geo-country change (!1150), the cron fixes (!1154), the vacuum fix (!1180), and the release-check fix for the new writeback service (!1182, without which this release would have jammed exactly like the 11-11 one), plus other teams' merged work.
2. The :free listing done properly, with admission tiers (platform-4a's combined MR, replacing !1149 and !1143).
3. !1152: Gemini Decisions.
4. !1145: fast rollback (shipped off), then its index, a staging rehearsal, and colors turned on in production.