The scientific method is a debugging technique

There is a moment in every serious production incident where the room splits into two kinds of engineer.

The first kind starts proposing fixes. The second kind starts proposing tests. In thirty years I have never seen the first group resolve an unfamiliar problem faster, and I have watched the second group solve things nobody expected them to.

Guessing dressed as expertise

The trap is that experience makes guessing look like diagnosis. When you have seen five hundred outages, pattern recognition fires immediately: this smells like connection pool exhaustion. Often you are right, and being right quickly is genuinely valuable.

But the failure mode is severe. A confident wrong hypothesis from a senior engineer will absorb the whole room's attention for an hour, because nobody junior is going to challenge it. Seniority converts a guess into a plan without anyone deciding to.

The discipline that protects against this is embarrassingly old:

1. State the hypothesis out loud, precisely enough to be wrong
2. State what you would observe if it were true
3. State what you would observe if it were false
4. Go and look

Step three is the one everybody skips, and it is the one that does all the work.

Precision is the whole game

"The database is slow" is not a hypothesis. It cannot be falsified, so it cannot be eliminated, so the investigation will keep circling back to it all night.

Compare:

If connection pool exhaustion is the cause, then active connections should sit at exactly the configured maximum of 40 during the incident window, and p99 latency should rise before CPU does, not after.

Now you have something. Two observations, either of which can kill the idea in ninety seconds. If connections peaked at 31, you have eliminated a whole branch of the search space — and eliminating branches is the only thing that reliably converges.

Changing one thing

The other borrowed discipline is the control. Under pressure, teams apply three mitigations at once: restart the service, raise the pool size, and fail over the region. The incident ends, and nobody learns anything.

You now have a system that recovered for an unknown reason, which means you have a system that will do this again on a date you cannot predict. The outage cost you the money; skipping the control cost you the lesson.

I understand the pressure. Sometimes the right answer genuinely is to throw everything at it and investigate later. But that should be a stated decision — "we are prioritising recovery over diagnosis, and we will reproduce this in staging tomorrow" — not something that happens by default.

Where science and engineering part ways

Science is willing to say we do not know yet, and take another decade. Engineering has a release on Thursday. That tension is real and I do not think it resolves cleanly.

What I have settled on is a rough rule about the cost of being wrong. Where a wrong guess is cheap and reversible — a config flag, a feature toggle — guess freely and let production teach you. Where it is expensive or irreversible — a schema change, a data migration, a security boundary — pay for the experiment.

Thirty years in, the most useful thing I learned was not a technology. It was the willingness to say "I think X, and here is how we would know I am wrong" — out loud, in front of people, before checking.

That sentence has saved me more hours than every debugger I have ever used.