Three Phases of Online Data Movement, Read as Three Invariants

Every online migration diagram I have seen looks the same. Three arrows between a small box and a bigger box, sometimes a callout that says "feature flag". The talks treat the three arrows as a sequence, but rarely as a sequence of things I can actually assert in production. I spent a few evenings going back through Stripe's writeups on DocDB — the engineering post on online migrations, plus the QCon San Francisco talk on petabyte-scale tenant moves — alongside the PlanetScale piece on rollback levels and the gh-ost and Vitess cutover docs. The pattern these sources keep landing on is the same. The substance I keep missing in my own designs is not the phases. It is what each phase is for.

That reframing is the takeaway I want to write down: an online data move is three phases stacked on top of three invariants, not one big switchover. Each invariant has a metric I can graph. Each phase ends when that metric crosses a gate I picked in advance. Each transition has a rollback that costs less than the migration step it ends. The work is in keeping each invariant observable, not in the cutover.

Here is what each phase is asking me to prove.

Phase one: dual-write with a freshness invariant

The first phase writes the same record to the old store and the new store on every mutation. The naïve framing of dual-write — "just write to both" — hides the part that breaks first. Without a per-record version stamp, I have no way to tell which copy is current when they disagree, and they will disagree. The two writes are not atomic. They cannot be, unless I am willing to pay for a cross-store transaction, and if I were, I would not be in this migration in the first place.

The invariant the version stamp lets me assert is freshness. I pick a budget — for example, the new store is no more than N seconds behind the old for any record — and I emit a metric per write: the gap between the old-store commit timestamp and the new-store commit timestamp for the same primary key. A p99 line on that gap is the gate. If it stays under the budget across a full diurnal cycle and a few traffic peaks, dual-write is doing what I want.

What "doing what I want" actually means is narrower than it sounds. Dual-write does not prove the new store is correct. It only proves that new records are landing in both stores in a bounded window. Historical records — everything written before the dual-write switch flipped — are still only in the old store. The freshness gate is necessary, but the only thing it gates is the next phase.

Skipping the version stamp is the failure I see most in writeups. Shops flip dual-write on, then a week later cannot explain why a record present in both stores has different values, because there is no ordering between the two writes and no way to reconstruct one. The version stamp does not have to be a monotonic clock; a hybrid logical clock or even the old store's commit LSN works, as long as a reader on the new side can compare two stamps for the same key and pick a winner.

The rollback at this phase is the cheapest of the three. Stop writing to the new store. Reads still go to the old store. Nothing the application has seen is different. The new store is allowed to be wrong; nobody is reading from it yet.

Phase two: backfill behind a shadow-read divergence budget

Phase two does two things in parallel. A backfill job copies the historical rows the dual-write missed. A shadow read fans every production read to both stores and compares results before returning the old-store answer. The user sees the old answer; the migration sees the diff.

The invariant here is the shadow-divergence rate. I pick a budget — for example, fewer than 0.01% of compared reads diverge in a rolling 24-hour window — and the gate is the cumulative rate staying under that budget for some number of windows back to back. Two clean windows is comfortable for small services. Stripe's writeups for petabyte tenants point at clean windows measured in days rather than hours, which sounds long until I think about the cost of a missed divergence at that scale.

What "diverge" means is the part I keep getting wrong on first design. The semantic difference between equal byte sequences and equal records is where the bugs live. Engineers running shadow reads have found this the hard way more than once. The published reports are explicit: timezones behaved differently between stores, NULL columns were filled with defaults on the way in, and sort order changed by a tiebreaker the new store did not preserve. None of those would have tripped a row-count or checksum check. All of them showed up as a diff in a Scientist-style comparator that knew the schema. GitHub's Scientist library is the original reference for this pattern — it runs both code paths inside the request, logs divergences, and returns the control answer to the user, which is exactly the property I need to keep latency unaffected during validation.

The shadow read has to run inside the read path the user actually takes. A separate offline diff job will not catch ordering and tiebreaker bugs, because it does not see the read context — the LIMIT, the cursor, the predicate plan. Running the diff in the request path costs a little p99 (one extra read in parallel), but it catches the differences that matter to users. From my reading, that latency cost is the main reason shops want to skip this phase. The cost they pay for skipping it is a cutover where they discover bugs in production with no rollback faster than re-running the backfill from scratch.

The backfill job that runs alongside the shadow read has its own properties to enforce. It has to be idempotent — a retried batch should leave the new store in the same state as a successful single run. It has to be throttled against a control signal on the new store, not a fixed rate; the pattern I borrow from Vitess's tablet throttler is to slow or pause when replication lag or read latency on the target goes above a threshold. A backfill that overruns the target is its own incident, separate from the migration.

The rollback at this phase is still cheap. Turn off shadow reads. Turn off the backfill. The old store is still authoritative for reads. The new store can stay populated or be wiped; nothing user-visible breaks either way.

Before walking through the cutover, it helps to see the three invariants stacked across time.

​

Phase three: cutover with a reverse dual-write cooling window

Phase three is where most diagrams end and most incidents start. The naïve framing is "flip reads to the new store". The version I write down in my notes is closer to: shift reads gradually, and keep writes flowing to the old store for the duration of a rollback budget I pick before the shift starts.

The shift is gradual because nothing else gives me a small enough blast radius. Stripe's published rollout for read switches is the canonical shape: 1%, then 10%, then 50%, then 100%, watching error rates between each step. The percentage cohorts and cohort selection — random, tenant ID, or hash-of-record — depend on the workload, but the property is the same: any error introduced by switching reads is bounded by the cohort size at the time it surfaces.

The invariant during cutover is the rollback budget itself. While reads are shifting to the new store, writes keep going to both stores via reverse dual-write: new store first, old store asynchronously. As long as the old store stays current within the freshness budget I set in phase one, the rollback is a single switch — point reads back at the old store, ignore the new one. The metric that gates this phase is the reverse-direction freshness gap: how far behind is the old store falling now that it is no longer the primary? If that gap blows past my budget, my rollback path is shrinking, and I have to either fix it or accept that the rollback is no longer an option.

The cooling window — the time I keep writing to both stores after 100% of reads are on the new store — is the part I always want to shorten and always regret shortening. The patterns I read converge on at least one full business cycle of clean operation before reverse dual-write turns off. For a payments service, that is a month-end and a quarter-end. For an analytics workload with monthly reports, it is two months. The cost of keeping reverse dual-write running is small. The cost of dropping it and finding a bug a week later is unbounded; the only path back to the old store is then a reverse migration from scratch.

Vitess's MoveTables.SwitchTraffic is a useful baseline for what the cutover actually does under the hood when a vendor implements it. It pauses writes on the source primary briefly, waits for the target to catch up, then routes writes to the target and adds the source to a denylist. The brief pause is the unavoidable cost of getting an atomic switch on writes; "online" tooling does not eliminate it, only minimizes it. gh-ost's MySQL cutover takes the same shape — an atomic two-step rename behind a brief metadata lock, with the original table still intact if the swap times out. Both of those are read- and write-cutover combined into a single atomic step. The three-phase pattern separates them, which is what gives the slow-roll on reads room to work.

Where a phase deserves to be skipped

Not every move needs three phases. Knowing when to drop one is more useful than running all three by default.

Skip phase one (dual-write) when the new store can be built from a change stream the old store already emits. With CDC or a transactional outbox running into the new store with bounded lag, dual-write is redundant — the change stream is the dual-write, with the bonus of being ordered. The freshness invariant is now "CDC lag below N seconds", which I am probably already graphing for other reasons. The trade-off is that I have given up the ability to write directly to the new store during the migration, which makes phase three's reverse dual-write awkward unless I add a temporary direct write path back.

Skip phase two (shadow reads) when the new store is a like-for-like replica running the same engine on the same schema. A shard split between two homogeneous Postgres clusters does not need a per-request semantic diff — replication, row counts, and checksums are enough. The moment the engines or schemas differ — going from a row store to a document store, switching JSON encodings, changing collations — shadow reads stop being optional. Sort order, NULL handling, and case folding are the corners where like-for-like assumptions break.

Skip phase three (gradual cutover) only when the cohort granularity is wrong. Some moves cannot be split: a single global counter, a singleton service-wide config, a leader-elected coordinator. For those, the cutover is a single switch with a maintenance window, and the three-phase pattern is the wrong tool. Picking it up anyway turns the migration into theater — a feature flag that flips for everyone at once is not a gradual rollout.

When the right answer is a maintenance window

The contrarian case I keep coming back to is the smallest one. Take the downtime. Reading the gh-ost limitations doc and the pgroll rollback-levels post side by side, the maintenance-window option is still the cheapest for some shapes of work. A 30-minute window in the middle of a Sunday for a low-throughput internal system is cheaper than three weeks of dual-write infrastructure plus the shadow-read diff cost plus the operational attention.

The honest question I ask before committing to the three-phase pattern is the SLO question: how much error budget do I have to spend on this migration, and over how long? If the answer is "not a single minute of degraded service is acceptable", the three phases earn their cost. If the answer is "an hour in the maintenance window I already have is fine", the three phases are over-engineering, and over-engineering a migration is its own incident class. Every additional moving part — the dual-write path, the shadow-read comparator, the backfill throttler, the cutover ramp, the reverse dual-write — is code that has to keep working under load, and any of it can be the thing that breaks instead of the migration it was supposed to protect.

What I am taking back to my notes

The pattern in one line: name each phase after its invariant, graph the metric, and pick the gate before the phase starts.

A short action list I am keeping:

• Treat dual-write as a freshness contract, not a copy job. Add a per-record version stamp before flipping it on.
• Run shadow reads inside the request path with a schema-aware comparator. Row-level checksums alone will miss the bugs that bite.
• Throttle the backfill against a target-side signal, not a fixed rate. An overrun backfill is a separate incident from the migration.
• Let reverse dual-write run for a full business cycle after the read switch. Shortening the cooling window is the rollback I lose first.
• Pick the rollback budget for each phase before the phase starts. If the metric blows the budget mid-phase, stop the migration. Do not raise the budget.
• Question the migration before designing it. A maintenance window is a legitimate answer and often the cheapest one.

When to reach for the three-phase pattern: high-throughput services with a tight SLO, heterogeneous source and target stores, or tenant-scale moves where per-cohort rollout is the only way to bound the blast radius. When to avoid it: low-throughput systems with available maintenance windows, like-for-like replica moves that need only standard replication, and any move where the cohort granularity is wrong for a gradual traffic shift.

The three arrows on the diagram are not the work. The three metrics under them are.