The Lock That Couldn't Hold Black Friday: How the Saga Pattern Killed Two-Phase Commit

12:01 AM, Black Friday

Somewhere in a Virginia data center, a payment coordinator service is holding its breath. A shopper in Ohio just clicked "Place Order" on a pair of sneakers. That single click fans out across six services: cart, inventory, payments, fraud detection, shipping, and email. In the old world — the world of relational databases and ACID guarantees — this would be one transaction. Commit or rollback. Clean.

But these six services don't share a database. They don't even share a company org chart. Inventory is owned by a team in Seattle running PostgreSQL. Payments is a team in Dublin running a proprietary ledger system. Shipping outsources to a third-party API that doesn't know what a "transaction" even means in your context.

And yet, somehow, the order has to be atomic. Either the sneakers get reserved, charged, and shipped — or none of it happens. This is the exact problem that, if solved badly, turns Black Friday into a trending topic for all the wrong reasons.

The Protocol That Promised Everything

In the 1980s, database researchers thought they'd solved this forever with Two-Phase Commit (2PC). The idea is elegant on a whiteboard: a coordinator asks every participant, "Are you ready to commit?" Each participant locks its resources and votes yes or no. If everyone votes yes, the coordinator says "commit" and everyone finalizes. If anyone votes no, everyone rolls back.

It's beautiful. It's also a loaded gun pointed at your own uptime.

Here's the problem nobody wants on a Black Friday war-room call: every participant holds its locks from the moment it votes "yes" until the coordinator's final word arrives. If the shipping service — the one hitting a flaky third-party carrier API — takes six seconds to respond instead of sixty milliseconds, every other service in that transaction sits there, locked, unable to serve any other customer touching that inventory row or that payment account.

Multiply that by ten thousand simultaneous checkouts and you don't have a slow request. You have a cascading pileup. And if the coordinator itself crashes mid-protocol — the single brain making every blocking decision — every participant is stuck in "uncertain" limbo, locks held, nobody able to safely commit or abort. This is literally called the blocking problem in distributed systems literature, and it's why 2PC, despite being taught in every database course since the 1980s, is functionally dead in modern microservice architecture. It assumes a single administrative domain, homogeneous infrastructure, and tight coupling — three things that do not exist when your checkout flow spans six teams and six different databases.

The 1987 Paper Nobody Was Ready For

The answer had actually been sitting in print since 1987, buried in a paper by Princeton's Hector Garcia-Molina and Kenneth Salem, titled simply "Sagas." They weren't thinking about microservices — that word didn't exist yet. They were thinking about long-running database transactions that were too large to lock efficiently, like a travel booking system reserving flights, hotels, and cars.

Their insight: stop trying to lock everything at once. Instead, break the big transaction into a sequence of local transactions, each one committing independently and immediately. If something later in the chain fails, you don't roll back the way a database rolls back — you can't, the earlier transactions already committed and released their locks. Instead, you run a compensating transaction for each completed step, undoing its real-world effect.

It's not rollback. It's semantic undo. You can't un-ring a bell, but you can apologize, refund, and re-stock. That distinction — compensation instead of rollback — is the entire philosophical pivot that let distributed systems abandon ACID at the system level without descending into chaos.

Anatomy of a Saga: The Sneaker Order

Let's run the actual order through it. The saga looks like this:

1. payment-charge — charge the customer's card
2. inventory-reserve — decrement stock for the sneakers
3. shipment-create — hand off to the carrier

Each step commits locally and immediately — no locks held across services. Now suppose shipment-create fails because the carrier's API times out. The saga doesn't panic. It walks backward, firing compensating actions in reverse order:

• inventory-release — put the sneakers back in stock
• payment-refund — return the customer's money

Notice: payment-refund is not the same operation as "never charging the card." The customer sees a charge and then a refund on their statement. That's semantic compensation, not true rollback — and it's a feature, not a bug, of how sagas work. You're not pretending the charge never happened. You're counterbalancing it.

Two Flavors: Choreography vs. Orchestration

The Garcia-Molina paper described the logic, but implementing it at Amazon or Uber scale split into two schools of thought.

Choreography has no central brain. Each service listens for events and reacts. Payment service charges the card, emits PaymentCompleted. Inventory service hears it, reserves stock, emits InventoryReserved. Shipping hears that, tries to create the shipment — and if it fails, emits ShipmentFailed, which inventory and payment independently listen for and compensate themselves. It's elegant for two or three services: no single point of failure, teams can evolve independently, and there's no bottleneck service to scale. But debugging it is like reconstructing a car crash from scattered witness statements — there's no one place to look at "the state of the order." Past three or four services, choreography turns into what engineers at companies like Uber and Netflix started calling "callback hell for distributed systems."

Orchestration puts a dedicated saga orchestrator in charge. It explicitly calls payment, waits, calls inventory, waits, calls shipping, and if shipping fails, it explicitly calls the compensations in order. This is centralized, which makes it easy to monitor, easy to visualize as a state machine, and far easier to onboard a new engineer onto. The tradeoff is the orchestrator becomes a critical piece of infrastructure — if it's not built for high availability and replay, it becomes the very bottleneck 2PC's coordinator was.

The rule of thumb that emerged across the industry: choreography for two or three services with simple linear flows, orchestration for five-plus steps with branching logic, retries, and timeouts. This is why Netflix built Conductor, an open-sourced orchestration engine, to manage its content-processing pipelines. It's why Uber built Cadence — later spun out and rebuilt as Temporal — a workflow engine that doesn't just run sagas but can replay their entire execution history from an event log if a worker crashes mid-flight. AWS customers reach for Step Functions to do the same thing with less operational overhead, and companies in regulated industries lean on Camunda Zeebe or the Axon Framework when they need auditable, versioned workflow definitions.

The Gotchas Nobody Puts in the Blog Post

Sagas trade one set of hard problems for another, and the teams that get burned are the ones who think compensation logic is an afterthought.

First: compensating actions must be idempotent. Networks retry. If your orchestrator thinks payment-refund timed out and fires it twice, you'd better not refund the customer twice.

Second: out-of-order events. In choreography especially, a ShipmentFailed event might arrive after a service has already moved on to handle a newer order on the same inventory row. Every handler needs to check: is this event still relevant to my current state?

Third: the saga deadlock problem — two sagas each waiting on a resource the other holds, not through locks, but through business logic (Saga A can't finish until Saga B releases inventory that Saga B is waiting on Saga A to release). This requires careful resource design, often solved by reserving optimistically rather than requesting pessimistically.

Fourth, and most unglamorous: the timeout problem. What happens when the shipping service simply goes silent — no success, no failure, nothing? A saga without explicit timeout-and-compensate logic will hang forever, which is exactly the blocking failure mode sagas were invented to avoid. Production saga implementations always define a timeout per step, after which the orchestrator assumes failure and runs compensations.

Why It Lives Next to Event Sourcing

It's not a coincidence that sagas, event sourcing, and CQRS tend to show up in the same architecture diagrams. If every state change is already an immutable event in a log — which is what event sourcing gives you — then a saga orchestrator can simply replay that log to recover its exact position after a crash. Temporal's entire value proposition is built on this: your workflow code looks like a simple sequential function, but underneath, every step is durably logged, so a server restart mid-saga doesn't lose state — it just resumes from the last recorded event.

The Legacy

Two-Phase Commit wasn't a bad idea — it was a correct idea for a world that no longer exists, a world of one database, one administrator, one blast radius. Microservices shattered that world on purpose, trading a single point of truth for independent, scalable, team-owned services. Garcia-Molina and Salem's 1987 paper, written years before anyone said "microservice" out loud, quietly provided the only sane answer: stop trying to lock the whole system. Commit locally, compensate honestly, and accept that "correct" in a distributed system doesn't mean "instantaneously consistent" — it means "eventually, provably consistent, with a clear undo path for every step." That's the trade every checkout button on the internet is silently making, every single time you click buy.