Motion Is Easy to Mistake for Mitigation

It is easy to confuse motion with mitigation. A team can be adding logs, trying to reproduce the issue, and preparing another release, and all of that can be useful work that does nothing to reduce what the customer is going through. Everything happening on the bridge can be necessary and still leave the customer exactly where they were when the call started.

When the product is broken in production, the first question is "Can we safely stop the bleeding?" Whether anyone is close to understanding it comes second.

Why It Feels Reasonable on the Bridge

None of this is obvious in the moment. The team is under pressure, and they may believe the fix is close. They may worry that rolling back will make things worse, or not trust the deployment pipeline enough to know whether the previous version is safe, or realize halfway through the call that nobody has ever tried to reverse the database migration that shipped with this release, and even a team that invested in feature flags can find out that none of them isolates the broken behavior.

So they say things that sound responsible and engineering-minded. They'd rather fail forward, they can't reproduce it in lower environments, and they don't want to roll back until they know exactly what happened. Sometimes that's all true. It can also cover for the actual issue, which is that nobody built the system to recover cleanly.

When Fail Forward Becomes a Reflex

Fail-forward has its place, and there are cases where a forward fix is safer than a rollback. It turns dangerous when it stops being a judgment call and becomes a reflex, especially when it's used to justify leaving customers broken while engineering searches for certainty.

A lot of that instinct comes from a world where releases were expensive. You shipped rarely, rollbacks were scary, and deployment was a ceremony, so pushing one more fix made sense because every release already hurt. With a modern CI/CD pipeline that thinking is legacy baggage. If releases are cheap, rollbacks should be cheap too, and a team whose rollbacks are still expensive has found the first thing to fix.

If You Can Only Understand Production in Production

"We can't reproduce it outside production" should start a conversation instead of ending one. It usually points to missing test data, thin logging, or configuration that has drifted between environments, and sometimes to a system that only behaves correctly under production load.

Not every production defect can be reproduced somewhere safe. Systems are messy, and some behavior only shows up at scale. But a team that keeps needing production to understand production has an engineering hygiene problem, and the organization has been borrowing confidence from the customer to cover it.

Shipping a Better Microscope

There's a particularly revealing version of this, where instead of mitigating, the team ships a new release that adds instrumentation to help find the problem. Sometimes that's necessary. Be clear about what it means, though. The customer is still absorbing the failure while the team gets a better view of it, which turns production into a microscope with the customer under the glass.

A release that only helps you understand the outage tells you the system was too opaque when it mattered. Instrumentation releases have their uses, but they belong near the bottom of the incident playbook, after every move that would have protected the customer.

The Bill You Never See

Engineering teams optimize for the costs they can measure, like time on the bridge, mean time to recovery, and incident counts. Some of the most expensive damage never shows up in any of them. Support volume climbs, and the account team has to explain what happened. Renewals get harder, and executives start asking whether the platform is stable. Some customers build workarounds, and some stop trusting the product at exactly the moment they needed it to work.

That damage is hard to act on precisely because it's hard to see. Customer confidence is expensive because you usually do not get a clean invoice for losing it.

Mitigation Before Diagnosis

"Always roll back" is too simplistic. The better rule is that when production is harming customers, mitigation comes before diagnosis. Root cause and the permanent fix still matter, and the order matters more than teams admit.

A mature production system gives you moves when something breaks. You can roll back the release, hit the kill switch on a feature or an integration, route traffic away, or fall back to a known-good path, and then debug with the pressure off.

The Customer Is Not Your Debugger

This is what boring software looks like during an incident. Things still go wrong, and when they do, the customer doesn't have to sit there while engineering discovers how the system actually works.

You can hear the difference in how a team talks on the bridge. A weaker team says, "We are still trying to figure out what happened." A stronger one says, "We have stopped the customer impact, and now we are figuring out what happened." It's the same incident, run by two teams who disagree about whose problem it is.

A mature engineering organization shows what it is made of by how quickly it can protect customers once production starts lying, and rollback, kill switches, and test environments that behave like production all exist so that production isn't where the team finally learns the truth. The next time a bridge call spends its second hour almost having the answer, somebody on it should ask what the customer is seeing right now.