A p99 that only got worse when we added capacity
We doubled the fleet and the p99 went up by 40ms. That sentence sat in our incident channel for two days before anyone could explain it, and the explanation turned out to be a good one, so here it is written down.
The setup
A read path: stateless API pods, a connection pool per pod, and a Postgres primary with two replicas. Reads go to replicas through a pooler. Median 4ms, p99 around 60ms at 12k reads/sec, which we were happy with.
Traffic forecasts said 20k/sec by April, so we went from 40 pods to 80. Median stayed flat. p99 went from 60ms to 101ms within an hour of the rollout, and stayed there.
What we checked first, and why it was all wrong
- Replica CPU: 31%, flat before and after. Not saturation.
- Query plans: identical, verified with auto_explain on the slow samples.
- Network: no retransmits, no change in RTT between availability zones.
- GC pauses in the API tier: unchanged, and too small to account for 40ms anyway.
Every dashboard we owned said nothing had changed. The thing that had changed was not on a dashboard: the number of connections.
The actual cause
Each pod opened a pool of 20 connections at startup and kept them warm. At 40 pods that is 800 connections into the pooler; at 80 pods it is 1,600, against a pooler configured for 900 server-side slots. Past that ceiling the pooler queues, and its queue is FIFO with no fairness across clients.
So the median request still found a free slot immediately — plenty were free most of the time. But the unlucky tail now waited behind a queue that had not existed at all a week earlier. More capacity in the stateless tier had made the stateful tier’s contention worse. Obvious in hindsight, invisible in the metrics we had.
Adding stateless capacity is only free until it multiplies your connection count against something that is not stateless.
The fix, in order of how much it helped
- Dropped the per-pod pool from 20 to 6, sized from measured concurrency rather than a number someone typed in 2021. p99 back to 58ms immediately.
- Added pool-wait time as a first-class client-side metric — this is the graph that would have found it in ten minutes.
- Set the pooler’s queue timeout to 250ms so we fail fast and shed load instead of holding requests the caller has already given up on.
- Wrote a capacity check into the deploy pipeline: pods × pool size must stay under 70% of pooler slots, or the deploy warns.
Three months on, p99 at 19k reads/sec is 54ms on 80 pods. The number that mattered was never CPU. It was a multiplication we were not doing.