The Retry That Outlived Its Token: Temporal Fakes in Go
I once shipped a test that mocked a token as always valid, and it hid a retry that reached the provider after the token expired. These are my notes on rebuilding that test in Go with a temporal fake — a stateful provider that shares a clock with the code — plus testing/synctest, so a 30-second TTL runs in microseconds and the race finally turns red.
I have shipped a flaky test I am not proud of. It mocked the auth client to return a valid token, called the code under test, and asserted success. Green every time. The code it covered then dropped a duplicate-free payment in production because a retried request reached the auth provider after the token had expired. The test could not have caught it. The mock had no concept of expiry, and that gap was structural, not an oversight I could patch with one more assertion.
Thoughtworks put a name on what was missing. Technology Radar Vol 34 (April 2026) added temporal fakes to the Assess ring: fakes that hold an internal state machine and evolve over simulated time, instead of a static double that returns whatever the test scripted (Technology Radar Vol 34). Their example is an observability stack for GPU data centers, where you cannot overheat a real card to test a thermal alert. The backend version is more pedestrian and bites more often: TTL caches, lease renewals, token rotation, retry budgets. This is the post where I rewrite one of those tests honestly, in Go, using a stateful fake and the testing/synctest package that landed stable in Go 1.25.
The one thing to take away: a temporal fake shares a clock with the code under test and holds the dependency's state, so a token that is valid when you read it can expire by the time the retried call lands. That race is the exact failure a static mock cannot represent.
Why the static mock cannot see the bug
Walk the failure in slow motion. A worker fetches a token with a 30-second TTL. It makes a call. The upstream returns a transient 503, so the worker backs off and retries. The backoff is exponential: 10 seconds, then 20. The third attempt lands at the 30-second mark. The token expired one instant earlier. The provider returns 401. If the retry loop treats 401 as fatal, the request fails; if it treats it as transient, it spins until the budget is gone.
The mock in my original test answered the same way regardless of when the call arrived: token good, here is your 200. Time did not exist in its world. In Martin Fowler's test-double taxonomy a stub like that returns canned answers to the calls made during the test, and nothing else (Test Double). A stub that ignores the clock is faithful to a provider that never expires tokens. No such provider exists.
A fake is different in kind. Google's testing group frames it as a lightweight implementation that obeys the same contract as the real thing, just with a shortcut — an in-memory store instead of a database, say (Test Doubles at Google). The contract for a token provider includes expiry. A fake that honors that contract has to track when each token dies. Once it does, "valid when read, expired when the retry lands" stops being a scenario I have to remember to script and becomes something the fake produces on its own.
The clock is the seam, not the mock
The reason this used to be painful in Go is that exercising a 30-second TTL meant either sleeping 30 real seconds — slow, and flaky on a loaded CI box — or threading a fake Clock interface through every call site and every library that reads the time. The first is the test I had. The second is invasive enough that I kept not doing it.
Go 1.25 removed the dilemma. The testing/synctest package, experimental in 1.24 and stable in 1.25 (the experimental Run was deprecated in 1.25 and removed in Go 1.26 — use Test), runs a test function inside an isolated "bubble" (Testing concurrent code with testing/synctest). Inside the bubble the time package uses a fake clock that starts at midnight UTC on 2000-01-01. Time only advances when every goroutine in the bubble is durably blocked — on a time.Sleep, a channel created in the bubble, a sync.WaitGroup.Wait, and a short list of similar operations. When the only goroutine parks on time.Sleep(10 * time.Second), the runtime jumps the clock forward 10 seconds and wakes it. No wall-clock wait, no flakiness.
The elegant part for a temporal fake: my fake provider reads time.Now() too. If the fake lives inside the same bubble, it sees the same fake clock as the backoff sleeps. The worker's time.Sleep advances simulated time, and the fake's expiry check observes that advance for free. The clock is the shared seam, and I do not have to inject anything by hand.
The diagram below is what I sketch before writing the fake: the token's state over simulated time, with the retry attempts plotted on the same axis so the collision at t=30 is obvious.
The fake, the bug, and the test that catches it
Here is the whole thing in one file. It is a fake token provider with a two-state lifecycle (valid, expired) plus a small transient-failure counter, a retry loop with the bug baked in, and the test that pins the failure.
package tokenretry
import (
"errors"
"testing"
"testing/synctest"
"time"
)
var (
errExpired = errors.New("auth: token expired")
errTransient = errors.New("auth: upstream unavailable")
)
// fakeAuth is a temporal fake of a token provider. It holds a state
// machine — one token with an expiry instant — and answers each call
// against the clock at the moment the call lands, not a scripted reply.
type fakeAuth struct {
expiresAt time.Time
failsLeft int
}
// Issue mints a token valid for ttl from the current (fake) time.
func (a *fakeAuth) Issue(ttl time.Duration) string {
a.expiresAt = time.Now().Add(ttl)
return "tok-1"
}
// Call validates the token, then simulates a few transient upstream blips.
func (a *fakeAuth) Call(token string) error {
if !time.Now().Before(a.expiresAt) {
return errExpired
}
if a.failsLeft > 0 {
a.failsLeft--
return errTransient
}
return nil
}
// callWithRetry retries on transient errors with exponential backoff.
// The bug: it reads the token once and never rechecks expiry.
func callWithRetry(a *fakeAuth, token string, attempts int, backoff time.Duration) error {
var err error
for i := 0; i < attempts; i++ {
if err = a.Call(token); err == nil {
return nil
}
if !errors.Is(err, errTransient) {
return err
}
time.Sleep(backoff)
backoff *= 2
}
return err
}
func TestRetryOutlivesToken(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
a := &fakeAuth{failsLeft: 2}
tok := a.Issue(30 * time.Second)
err := callWithRetry(a, tok, 5, 10*time.Second)
if !errors.Is(err, errExpired) {
t.Fatalf("got %v, want errExpired", err)
}
})
}Run it with go test -run TestRetryOutlivesToken on Go 1.25 or newer.
A few lines carry the argument. Call checks time.Now().Before(a.expiresAt) first — that single comparison is the entire difference from a static stub, because it makes the answer depend on when the call happens. callWithRetry reads the token once and never revisits it; that is the production bug I am reproducing, not a contrivance. The test issues a 30-second token, lets the loop fail transiently twice, and the two backoff sleeps (10s then 20s) carry simulated time to exactly the expiry boundary. The third call lands at t=30 and returns errExpired.
The result that matters is the wall-clock time. On my run the test reported 0.00s while advancing 30 seconds of simulated time. With real timers it would block for 30 seconds and still be flaky on a slow runner. And the contrast with the mock is sharp: a stub returning transient, transient, nil would let callWithRetry succeed on the third attempt and the test would pass — green, and blind to the race. The temporal fake ties success to the clock, so the same loop fails the way it fails in production.
The fix, once the test is red against the real bug, is the easy part: cap the total retry budget below the TTL, or refresh the token before any attempt that would land inside a skew buffer — refresh when now >= expiresAt - skew rather than waiting for the hard boundary, which is the standard guidance for token-expiry races under clock skew (Nango on OAuth refresh concurrency). The point of the fake is that it makes the broken version observably broken first.
Where temporal fakes lie to you
The Radar names the trap in the same breath as the technique: a fake earns trust only while it stays faithful to the real dependency, and a drifted fake manufactures false confidence in a green pipeline. My fake models expiry as a clean instant. A real provider has clock skew, a grace window, refresh-token rotation that can invalidate the old token early, and 5xx responses that are not the transient blips I scripted. Each gap is a bug the fake will cheerfully fail to catch.
The defense is a contract test: one suite that runs against both the fake and the real provider, gating on the behavior both must share. Google's writing on test fidelity makes the same case — a fake without a contract is just a guess that compiles (Increase Test Fidelity By Avoiding Mocks). I treat the fake as the fast path and the contract test as the thing that keeps it honest.
Two boundaries kept me from over-reaching. First, synctest only advances time when goroutines are durably blocked, and the rules are specific: mutexes are not durably blocking, and real network or file I/O is not either. A fake that hides behind a real socket or blocks on a global mutex held outside the bubble will stall the clock or panic the bubble. The fake has to live in-process and communicate through bubbled channels or direct calls — which a temporal fake does anyway. Second, this technique is about correctness over simulated time, not resilience. It is a different tool from deterministic network simulation: a simulator like Tokio's turmoil drives the scheduler and the network to reproduce a partition race byte-for-byte, while a temporal fake models one dependency's state machine as the clock moves. I reach for the simulator when the bug lives in the network ordering, and the temporal fake when it lives in a dependency's lifecycle.
When to reach for this, and when not
- Use a temporal fake when correctness depends on when a call lands relative to a dependency's state: TTL cache expiry, lease renewal windows, token rotation, retry budgets that can outrun a deadline, backoff that can starve a timeout.
- Pair it with
testing/synctest(Go 1.25+) so the fake'stime.Now()and the code'stime.Sleepshare one fake clock, and a 30-second TTL test runs in microseconds. - Back it with a contract test against the real dependency, or the fake will drift into false confidence — the Radar's own warning.
- Reach for deterministic simulation instead when the bug is in network ordering or scheduler interleaving rather than a dependency's lifecycle.
- Skip it for stateless dependencies, or where a plain stub already tells the truth. Fidelity you do not need is maintenance you will resent.
The honest version of my original test is barely longer than the dishonest one. The difference is a single comparison against a shared clock — and that comparison is what turns "the mock said it was fine" into a test that fails the way production does.
Still here? You might enjoy this.
Nothing close enough — try a different angle?
Related Posts
Read-Your-Writes Is a Session Contract, Not a Database Setting
I added a read replica and within a day users reported edits that "didn't save" — they had saved, but the reads raced the replication stream and lost. These are my notes on why read-your-writes is a session contract rather than a replica setting, reproduced with a pinned 50 ms lag in Go, and the three ways to carry it — sticky routing, a GTID-style version token, and a bounded-staleness wait — compared on cost.
Catching a Retry Race with One Seed: Deterministic Simulation in Rust using turmoil
I had three flaky retry tests no one could reproduce on a laptop. I rewrote one in Rust on top of turmoil, Tokio's deterministic simulator, and a single 8-byte seed pinned the partition race byte-for-byte. These are my notes on what the seed actually controls, what leaks past it, and when deterministic simulation testing is worth the seam.
AckWait Is a Contract: How a 30-Second Default Took Down My JetStream Consumer
I lost an evening to a NATS JetStream pull consumer that doubled its work in production. The cause was three lines of ConsumerConfig I never wrote. These are my notes on what AckWait actually counts, why MaxDeliver = -1 is the silent footgun, and the 70-line Go contract I now ship on every JetStream consumer.