Your Developer Didn't Break Production. Your Process Did.
Sajidur Rahman Sajid
Software Developer
Something goes down. Checkout is failing, the dashboard is red, and someone is already scrolling the deploy log looking for a name.
They will find one. Somebody always pushed the change. But that name is the least interesting fact in the entire incident, and the moment a team treats it as the answer, the investigation stops one step short of anything useful.
The real question is not who pushed the change. It is why pushing that change was possible at all.
"Human error" is a starting point, not a verdict
Every mature safety-critical field figured this out decades ago. Aviation, medicine, and industrial engineering all stopped accepting "the operator made a mistake" as a root cause, because it explains nothing and prevents nothing. The next operator will make the same mistake, because the conditions that produced it are untouched.
Software is slower to catch on, partly because the cost of one outage is rarely a life, and partly because finding a person to blame is genuinely satisfying in a way that fixing a deploy pipeline is not.
But the logic holds. If a tired engineer at 5pm on a Friday can run one command and take down checkout, the interesting finding is not that they were tired. It is that the command existed, was reachable, and had nothing between it and production.
What the outage is actually telling you
Read the incident as a diagnostic of the process, not the person. Almost every production failure maps onto a small set of structural gaps.
| What happened | What it says about the process |
|---|---|
| One person deployed and nobody saw it | No review gate on the path to production |
| It worked locally and failed in prod | Staging does not resemble production |
| Recovery took hours | No rollback, or rollback is a manual runbook |
| Nobody noticed until a customer called | Alerting covers infrastructure, not user outcomes |
| The fix required one specific person | Knowledge lives in a head, not in the system |
| The same class of bug recurs | Postmortems produce discussion, not owned actions |
None of those rows have a name in them. That is the point. Every one is a decision somebody made, or failed to make, months before the incident.
The rollback question
If you want one question to assess a team's real maturity, it is this: how long does it take to get back to the last known good state, and who can do it?
The answers you hear usually fall into three tiers. Best case, anyone on call can trigger a rollback from a dashboard in under two minutes. Middle, there is a documented procedure and two or three people have run it. Worst, and most common, there is a procedure in a wiki that nobody has tested since the person who wrote it left.
An untested rollback is not a rollback. It is a hope with formatting.
What blame actually costs you
The practical argument against blame is not that it is unkind. It is that it makes outages longer and more frequent.
In a team where incidents end in someone being singled out, people stop volunteering information. Near misses go unreported. A risky change gets pushed quietly instead of flagged. When something does break, the first instinct is to work out whether it is provably yours before saying anything, and that hesitation is measured in minutes of downtime.
You end up with a team that is worse at detecting problems, slower at responding to them, and completely blind to the ones that almost happened. The incident count on the dashboard may even fall, which is the most dangerous outcome of all, because nothing improved except the reporting.
Blameless is not the same as unaccountable
This is where the idea usually gets misread, so be precise about it.
Blameless means nobody is punished for an honest mistake made inside the system as it was designed. It does not mean nothing is anyone's problem. Every postmortem should produce a short list of concrete changes, each with a named owner and a date, and those owners are accountable for delivering them.
The accountability moves from "who made the error" to "who is fixing the condition that allowed it." That is a much more productive place to put pressure, and it is the only version that actually reduces the next outage.
Genuine negligence, repeated after being addressed, is a separate management conversation. It is also rare, and treating every incident as though it might be that case is how teams end up with the silence problem above.
Five changes that move the needle
1. Make the safe path the easy path. If deploying correctly takes six steps and deploying carelessly takes one, people will occasionally take the one. Automate the six.
2. Make rollback boring. One command or one button, tested regularly, runnable by whoever is on call at 3am. Practise it when nothing is on fire.
3. Remove manual steps from production. Every manual step is a place a human is required to be perfect while tired. Those are the steps that fail.
4. Alert on user outcomes, not just infrastructure. CPU looks fine during most real outages. Checkout success rate does not.
5. End every postmortem with owned actions. Not "we should improve monitoring." A named person, a specific change, a date. Anything vaguer will not survive the next sprint.
