The Saga Pattern Is About Time, Not Services

On Bullpen, every buy goes through the same four steps: reserve the buying power, hand the order to the venue, book each fill as it arrives, and release whatever the order didn't use. The trouble starts when those steps stop arriving in order.

A buy order on Bullpen asks for 100 shares. Forty fill. The player hits cancel. A second fill of 25 was already executed when the cancel went out, but its message reaches Bullpen after the order shows "cancel requested". What does the ledger say now?

No single database transaction can answer that, because no single transaction lasts long enough to see it happen. The order lives for minutes; a transaction should live for milliseconds. That is the whole reason Part 4 needs the saga pattern, and the reason is time, not a new service or a second database. Bullpen still keeps every step of an order inside one atomic transaction in the trading schema, as Part 3 decided. What changes is that an order now crosses several of those transactions, and each one needs a defined way back, or a defined way forward when there is no way back.

This part follows the same decision history as Part 3, but only the three decisions that carry the argument get the full treatment. Everything else is in one table.

Why the saga pattern was in the plan from week one

The saga is older than microservices. Hector Garcia-Molina and Kenneth Salem described it in their 1987 SIGMOD paper for a problem inside one database: long-lived transactions that hold locks for so long that everything else waits. Their answer was to break the long transaction into a sequence of short ones, each committed on its own, each paired with a compensating transaction that semantically undoes it if a later step fails. The database never has to hold the whole thing open.

The microservices meaning came decades later. Chris Richardson's saga pattern uses the same idea to keep data consistent across services that each own a database, and that is the version most engineers learned. It is also the version that made me put an order saga into Bullpen's plan in week one: ADR-0002 named it as a split that would have to justify itself, Part 3 deferred the order lifecycle to this part, and the planned architecture for Part 4 still shows an Order Saga node, a separate workflow engine beside the trading app.

I expected the saga to be the first real split in Bullpen. Reading the original paper again changed that expectation before any code ran. The 1987 problem is Bullpen's problem: a business process that outlives a transaction. Whether it also needs a new runtime is a separate question, and Decision 1 is where the build answers it.

What one transaction can't hold

​Part 3's decision 3 put the order, its ledger entries and the buying-power reservation in one transaction. That still holds for every individual change. It cannot hold for the lifecycle, for three plain reasons. A Vercel Function on Hobby stops after 300 seconds. A Postgres transaction left open holds its row locks for as long as it lives, and its old snapshot keeps vacuum from cleaning up behind every other write. And a fill is not something Bullpen decides; it arrives, sometimes in pieces, sometimes late.

So the order becomes a small state machine, written down in ADR-0017. The rules, the decision record and the tests that check them landed in the same commit. I wrote the rules first while working, but the history does not show a commit with the rules alone, so I won't claim more than it shows.

Four rules matter more than the states themselves:

• A cancel request is not a cancel. The player asks; the cancel takes effect only when the execution side confirms it. Between the two, fills can still happen.
• A fill cannot be undone. If 40 shares filled before the cancel took effect, those 40 shares are the player's. The recovery for "partially filled, then cancelled" is not compensation. It is booking what filled, cancelling the rest and releasing only the unused part of the reservation.
• Every fill has an identity. A unique fill id means a duplicate delivery changes nothing.
• Every transition checks the version it read. The same conditional-update pattern Part 3 used for buying power, applied to the order row.

After every test run, the same invariants have to hold: buying power never negative; reserved plus spent plus available equals the starting cash; filled plus remaining plus cancelled quantity equals the ordered quantity; positions equal the sum of booked fills; and Part 3's reconciliation check still passes.

The picture below follows the order from the opening on one time axis. Watch the second fill: it is executed before the cancel takes effect but arrives after the cancel request, and that gap is the whole problem.

The vocabulary, applied once

Three terms are enough for this post, and each one maps to a choice Bullpen had to make.

Compensation versus recovery. A compensation is a new transaction that semantically reverses an earlier one, like releasing a reservation. Some steps have no compensation, like a fill. For those, the saga needs a recovery path that moves forward instead. Writing down which is which, per step, is where most of the design work sits.

Orchestration versus choreography. In an orchestrated saga, one coordinator tells each step what to do next and owns the state. In a choreographed saga, each participant reacts to events from the others and nobody owns the whole picture. Bullpen has one owner of orders and one database, so orchestration is the natural fit; the open question is only where the orchestrator lives.

The saga matrix. Software Architecture: The Hard Parts describes sagas along three dimensions: synchronous or asynchronous communication, atomic or eventual consistency, orchestrated or choreographed coordination. That gives eight combinations, and the book names each one. Bullpen's order lands on asynchronous, eventual and orchestrated, which the book calls the Parallel Saga: fills arrive when they arrive, the order converges to a final state rather than committing all at once, and one coordinator drives it. For errors in that shape, the book leans less on chains of compensating updates and more on a state machine that always knows where each transaction stands and corrects it through retries or a manual fix. That state machine is where Decision 1 starts.

Decision 1: where the saga runs

Conditions. Bullpen runs on Vercel Hobby with one Neon database. Four platform pieces could drive or time the lifecycle, and their limits shaped the options:

The sources are Workflows pricing and limits, Workflows retention, Queues concepts, cron job usage and Daemons limits. Two more conditions decided more than any number in that table. Postgres and a workflow engine cannot commit together, so anything outside the database needs a reliable handoff. And on Hobby a finished workflow run's payloads stay readable for one day, so Postgres has to hold everything needed to explain and recover an order on its own.

Options.

• A Postgres state machine. The order row is the state; every transition is a conditional update in one transaction; a sweep advances or flags orders that stopped moving.
• Vercel Workflows as the orchestrator. Each step is a 'use step' function that runs one transaction in trading; the workflow holds the sequence, retries and waits. Vercel documents a saga recipe where compensations are written explicitly and run in reverse; there is no built-in per-step rollback.
• Vercel Queues with choreography. Each step publishes the next event; consumers react. The state still lives in Postgres.

The state machine needs a sweep, and the sweep needs a clock that runs more than once a day. That sub-choice has its own options: a Vercel Daemon polling every few seconds, a delayed Queues message per order used as a timer, or a workflow sleep if Workflows wins.

Choice. The Postgres state machine, driven by a daemon. The daemon calls an authenticated sweep route every 5 seconds while any order is open, and backs off to every 60 minutes when none is. At that pace, a day with no orders keeps Neon awake about 2.3 hours, roughly $1.86 a month; that figure is computed, not read from Neon. The sweep itself is just more conditional updates: every step checks the order's state before it acts, so running it twice, late or after a crash changes nothing.

Why. It was the option I could build and measure without a new integration, and it passed every correctness test below on its own. A stopped daemon only delays recovery: a test stops it for days, and the next sweep finishes every order with the invariants intact. Its own cost is small. Locally, a sweep call takes about 2.1 ms of daemon CPU; at 5 seconds all month that is 518,400 calls and about 0.31 of the Hobby account's 10 Active CPU-hours.

I did not build Workflows, and I did not measure it. The one gain a workflow sleep could show here is Neon's idle cost, because a timer that fires only when an order needs it lets the database sleep. Measuring that needs a deployment with Workflows writing to Neon. It would also bring the handoff below, which is new risk for a baseline that already passed. So the conclusion is narrow: Part 4 did not need Workflows. It is not that Workflows would be worse.

The handoff is the part I underestimated on paper. If Workflows drove the order, the order row would commit in Postgres and the workflow would start in Vercel, and those are two separate commits. An order could commit and never start, or a retry could start it twice. The fix is an outbox: the start event is written to a table in the same transaction as the order, and a dispatcher sends it and marks it sent, idempotently. I wrote about why an outbox is a ledger and not a queue in my notes on the transactional outbox. A pure Postgres state machine has no handoff at all, and that ended up deciding as much as any number.

What would flip it. Cost, not correctness. A daemon that polls keeps Neon from scaling to zero. If the sweeper keeps the compute awake more than about 8 hours a day on average, which is about 61 CU-hours and $6.50 a month at 0.25 CU on Neon's Launch plan, a delayed message or a sleep that touches Postgres only when an order's timer fires becomes the cheaper driver. A second consumer of order events, or a step that has to wait for days, would flip it too.

Decision 2: the cancel that races a fill

Conditions. Fills arrive at least once and not necessarily in order. A cancel takes effect on the execution side, not when the player taps the button. The ledger has to be right after every interleaving, not just the likely ones.

Options.

• Cancel wins. Any fill that arrives after "cancel requested" is rejected.
• Last write wins. Whatever reaches the database last decides the final state.
• Execution time decides. A fill executed before the cancel took effect is booked, whenever it arrives; the cancel then applies only to what remained.

Choice. Execution time decides. Rejecting a real fill would leave the player's position out of step with what the execution side did, and last-write-wins makes correctness depend on network timing.

Why. This option is the only one where the invariants can hold in every interleaving, and it reuses Part 3's mechanism instead of adding one. Each transition is a conditional update on the order's version; each fill is inserted under a unique fill id, so a second delivery of the same fill finds the row already there and changes nothing. I treated the deduplication as more than a key, for the reasons in my notes on idempotency as a protocol: the duplicate also has to arrive while the first delivery is still running.

The simulator's ordering rules are in the same ADR. Every fill has an id and a venue time, and arrives at least once, in any order. The venue confirms a cancel with an effective time, and every fill executed before that time is owed whenever it arrives. The unused reservation is released only after the effective time plus a 30-second window for late fills, and the sweep retries the release until it succeeds.

The race ran two ways, against a local Postgres. An exhaustive run tried 8,346 distinct delivery orders of fills, duplicates, cancel requests and cancel confirmations. A random run tried 1,500 seeds with events in flight at the same time, injecting 1,985 crashes and 915 duplicate deliveries. Both had zero invariant failures. To check that the test could fail at all, I changed one line so the lifecycle dropped any fill arriving after a cancel: 4,330 of the 8,346 exhaustive runs failed, and I reverted the change.

What the runs don't cover: one account and one small order per run, at most four fills, a delay longer than the 30-second window, many accounts contending at once, a real exchange, or Neon itself. And one thing they haven't explained yet: on October 26, one CI run of this test stopped on a Postgres error, new multixact has more than one updating member. No account check failed, and the re-run passed, but I haven't found the cause. It goes on the list for Part 5's load test.

What would flip it. An execution venue that reports cancels and fills without a shared execution time. Then "executed before the cancel took effect" can't be decided, and the order would need the venue's own final report before settling.

Decision 3: where fills come from

Conditions. Part 4 needs fills that arrive late, in pieces and occasionally twice, on demand, in tests that can be repeated. Alpaca's paper-trading environment documents random-sized partial fills about 10% of the time an order is eligible to fill, which Part 1 already used as the bar for realism.

Options.

• Call Alpaca's paper-trading API.
• Write a simulator based on Alpaca's documented behaviour.

Choice. A deterministic simulator behind an adapter. It is seeded, it can inject a delay or a crash at a named boundary, and it can force the exact race from the opening. It is based on Alpaca's documented behaviour, not a copy of it; the adapter leaves room for a real Alpaca run later.

Why. Correctness tests need to reproduce a failure, and an external sandbox can't promise the same interleaving twice. Adding a provider would also have added a new dependency to the one part of Bullpen that has to be provably right.

What would flip it. A question only the real venue can answer, such as how its cancel confirmations are ordered relative to fills. That is a Part 5 or later experiment.

What I broke on purpose

Each failure below has a test in the repository and an entry in the Part 4 report. A happy path proves little here; each of these has to leave the account correct.

A rollback to an earlier deployment and the deletion of a deployment are different failures from a crash. Vercel's Workflows concepts say runs stay pinned to the deployment they started on, and runs pinned to a deleted deployment never finish on their own. I didn't build Workflows, so those two stay as reading, not testing.

The failing release is the one I'd show first. Cloudflare's design, from their June post on Workflows rollbacks, stops the remaining rollbacks and errors the run when a rollback fails. Bullpen can't stop: the money is still reserved, and the player needs it back. So the order stays cancelled, the unused amount stays counted as reserved, which keeps every invariant true, and the sweep tries again on every pass. On the third failure the order is flagged. The next successful sweep released all 650 and cleared the flag. What's missing is the alert: today the flag is a column, and nothing pages anyone.

What production found that the tests didn't

Every test above passed before the first real order. Production still found three problems in the first two days, and all three had the same shape: a test supplied something production never would.

• Sign-in. The tests signed their own token in the format my code expected. Neon's SDK keeps the session in a cookie signed with a secret, a different format, so every signed-in request came back 401. The fix takes the player from the SDK's own session check (ADR-0022), and the test now uses the SDK's real cookie.
• Migrations. Deploying the code and applying its migration were two manual steps. Twice in one day, production ran code whose tables didn't exist yet. Now a production build applies the migrations before it builds, and fails if it can't (ADR-0023).
• Fills. The simulated venue filled only at a price under 5 minutes old, and every order test seeded a price a few seconds old. Stock prices on Bullpen are end-of-day closes, hours old by the time anyone trades. My first three real orders, 90 AMZN, 12 V and 3 AAPL, waited 15 minutes and were cancelled by the sweep. The ledger was right: every reservation came back in full. The rule was wrong for the data. Now each price says whether it is live or end-of-day, and only a live price has the 5-minute limit (ADR-0024). After the fix went live on October 26, a market buy of 1 AMZN went from reserved to filled in the same second, at $248.00, the October 23 close, and the unused $12.40 of the reserve came back.

The last one is this post's argument showing up uninvited. The rule was about time, and the test's clock wasn't production's.

The smaller decisions

That last row is Part 3's lesson applied up front: the setting is read from the system before any number goes in the post.

What the build changed in the records

Four decision records carry the design. ADR-0017 is the order lifecycle as a Postgres state machine, and ADR-0018 is the sweeper, with ADR-0021 setting its idle wait to 60 minutes. ADR-0019 makes the leaderboard count reserved buying power as cash: without it, a player's equity dipped by the reserved amount while an order was open, which an equity check after every order test now catches. ADR-0020 is sign-in through Neon's managed auth. Production added three more, one for each problem above (ADR-0022 to ADR-0024), and ADR-0025 adds the previous close to the price contract for the trading screen.

Part 3's records stayed Proposed until the post went out. This time I accepted the first ones during the build, because ADR-0016 says only an Accepted decision may name a new node in the model, and the sweeper and Neon Auth had to appear in the Part 4 moment. ADR-0023 to ADR-0025 were accepted with this post.

The Part 4 moment in the architecture model is now the current one, and the explorer on the architecture page follows it. The Order Saga node that week one predicted shows as predicted and not built: there is no saga service, only steps inside the trading schema and a daemon that keeps them moving. The predictions-week-1 tag did not move.

The trading app is open for orders at bullpen-app.tiarebalbi.com. Everyone starts with 100,000 in play money. A signed-in player gets their equity and rank in the header, a watchlist with what they hold, a positions table with average cost and profit or loss, a market-buy ticket, and a step-by-step timeline of their open order with a cancel button. "Cancel requested" has its own words and mark, and never says the money is back; a cancelled order shows what filled and what was released. There is no sell, no limit order and no chart yet, because none of them has data behind it. They come with live prices in Part 5.

What Part 4 cost

I measured cost per order locally, over 40 market buys:

The daemon used about 37 seconds of CPU a day at the 5-second cadence, locally; that is not a Vercel Active CPU reading. Production has run since October 21, which is too short for a week's reading before this post. The daemon's Active CPU, Neon's CU-hours since orders exist, Neon Auth's cost on this plan and Vercel's usage go into the cost ledger when the week closes, and Part 5 reports them. Until then they are pending here, not estimated.

Where the time in a transaction comes from

Split the work where time forces you to, and keep each step atomic. Define the compensation, or the recovery path when an action cannot be undone, before you write the step.

The saga in Bullpen came from the clock, not from the org chart or the deployment diagram. Before reaching for a saga in your own system, it is worth asking which one is yours: is the work long because it waits on the world, or only because it crosses a boundary you drew?