Queueing theory explains your outage

Here is a conversation I have had perhaps thirty times.

"The servers are only at 75% CPU, so we have headroom."

You do not have headroom. You are one small traffic increase away from a very bad afternoon, and the reason is a piece of mathematics that predates almost every technology in your stack.

The equation

For a simple queue, the average time a request spends waiting relates to utilisation like this:

W  ≈  S × ρ / (1 − ρ)
 
W = average wait time
S = average service time
ρ = utilisation (0 to 1)

The interesting part is the denominator. As utilisation approaches 1, 1 − ρ approaches zero, and wait time approaches infinity. Not linearly. Asymptotically.

Put numbers on it, with a service time of 100ms:

Going from 50% to 70% utilisation costs you 133ms. Going from 90% to 95% — the same five-point step you took happily earlier in the year — costs you a full second.

This is why the graph of your incident looks like a hockey stick, and why nothing appeared wrong until moments before everything was wrong. The system was not stable at 75% and then broken at 85%. It was always on this curve. You were simply on the flat part of it.

Little's Law, and the mistake it prevents

The companion result is even simpler, and it is the one I reach for most:

L = λ × W
 
L = items in the system
λ = arrival rate
W = time in the system

That is it. Three variables, no assumptions about distributions, true for any stable system you can point at.

Its practical use is sizing. A team once asked me to review a thread pool of 200 for a service handling 500 requests per second with a 40ms average response time. Little's Law says the average number of in-flight requests is 500 × 0.04 = 20. The pool was ten times oversized — which sounds harmless, until you realise that 200 threads all making a downstream call is how you turn your own service into a denial-of-service attack against a dependency that was correctly sized.

The pool was not headroom. It was a loaded weapon pointed at the database.

Why variance is the real enemy

The clean equation assumes exponential service times. Real systems are worse, because real workloads are mixed: a hundred requests taking 10ms and one taking 4 seconds have a pleasant average and a terrible experience.

This is the actual argument for separating workloads onto separate queues — not tidiness, but variance isolation. One slow tenant, one enormous report, one pathological query: in a shared queue, each of them taxes everybody behind it. The mathematics is unforgiving about this in a way that intuition simply is not.

Every capacity problem I have seen dressed up as a scaling problem was really a variance problem. Adding servers moves you left on the curve. Reducing variance changes the curve.

What to do on Monday

Three things, and none of them require new tooling:

• Target 60–70% utilisation for latency-sensitive services, and stop treating the gap as waste. You are buying the flat part of the curve.
• Size pools with Little's Law, then add a modest margin. If your pool is more than a few times λ × W, ask what you think it is protecting you from.
• Measure p99, not the mean. The mean is where queueing theory hides its bad news.

Sixty-five years old, four symbols, and it will predict your next outage better than most monitoring products. Mathematics is quietly the most practical subject on this site.