All posts
Long-form notes on software, distributed systems, and the craft of building. Shipping one a week.
The Retry That Outlived Its Token: Temporal Fakes in Go
I once shipped a test that mocked a token as always valid, and it hid a retry that reached the provider after the token expired. These are my notes on rebuilding that test in Go with a temporal fake — a stateful provider that shares a clock with the code — plus testing/synctest, so a 30-second TTL runs in microseconds and the race finally turns red.
The Spectator Trap: Staying in Control of AI-Assisted Development
My feeds are full of screen recordings of developers watching an agent write code — and I want to name that posture kindly: it is spectating, not productivity. Closing the line of thinking from Zero Token Architecture and The Handoff Is the Unit of Design, these are my notes on the control plane a developer should never leave: small parallel handoffs instead of accept-all, research and side-effect mapping automated ahead of implementation, and the Log4Shell-shaped warning about shipping code nobody understands — with the costs of the conscious handoff named as honestly as its benefits.
Forecasting Is Now a Library Call: Notes from Running TimesFM 2.5
Google's TimesFM 2.5 shrank to 200M parameters, topped GIFT-Eval zero-shot at release, and now sits behind BigQuery's AI.FORECAST — so I ran it myself on telemetry-shaped series. Zero-shot, it beat seasonal-naive by 15% in 0.6s per forecast on CPU. Then I added a 60% level shift one week before the forecast and the one-line baseline won by 1.5x — while the model's quantile band quietly widened 2.6x. These are my notes on what caught the attention, what to observe before trusting it, and the use cases where I would actually wire it in.
The Handoff Is the Unit of Design: Delegating to Agents Without Losing the System
Once agents write a meaningful share of the code, my output is no longer typed code — it is delegation decisions. These are my notes on the human discipline that makes that work: sizing every handoff to the review I can afford, the brief I hand over instead of big tasks, and the four habits that keep me connected to a system I am no longer typing into — from Bainbridge's 1983 ironies to a METR result that has since reversed its own sign.
Your Queue Won't Drain: The Arithmetic of Backlog Recovery
A healthy-looking consumer fleet can sit on a 3-million-message backlog that never shrinks. These are my notes on the arithmetic behind it: surplus = capacity − arrival, drain = backlog ÷ surplus, and why a fleet sized for steady-state has zero recovery capacity. I built a small TypeScript simulator to watch retry amplification park a correctly sized fleet in a metastable failure, and worked out when shedding beats draining.
Cache Stampedes Are a Capacity Problem, Not a Locking Problem
A distributed lock on every cache miss is the fix most posts reach for; in my own testing it is the one that turns a 400 ms blip into a queue collapse. These are my notes on treating stampedes as an upstream capacity budget — why 10% TTL jitter is a cargo-cult number, what XFetch actually buys, and the rubric I use to decide when a lock earns its keep.
Backpressure Is a Contract Every Caller Must Honor, Not a Setting
A bounded queue in one service does nothing if the caller above it keeps pushing. These are my notes on backpressure as an end-to-end contract: what the callee owes, what every caller owes back, and why the Azure OpenAI retry-amplification incident is what a broken clause looks like at hyperscale.
When a Zone Fails, Don't Let Your Cluster Rebalance Itself to Death
An availability zone drops and the cluster survives the outage — then hurts itself trying to heal. These are my notes from reading Uber's zone-resilient OpenSearch write-up against the allocation-awareness docs: why the re-replication reflex is the real danger, and how declaring every failure domain up front makes a cluster degrade to a stable yellow instead of a rebalance storm.
The Bug That Wouldn't Compile: Porting a Reordering Mistake to Hydro
I planted an arrival-order bug in a hand-rolled Rust fan-in aggregation, reproduced a 51% undercount with a seeded shuffle, then ported the same logic to Hydro 0.17-alpha. The unordered merge refused to compile my non-commutative fold — these are my notes on what that guarantee is, what manual_proof! only attests, and when the seam costs are worth paying.
Architecture as Code with Structurizr DSL: A C4 Workspace Both a Reviewer and an Agent Can Read
I committed a Structurizr DSL workspace next to a Kotlin/Spring billing service and wired ArchUnit into Gradle so CI fails the moment the code drifts from the C4 design. Here is the workspace.dsl, the test that backs it, and what changed when a coding agent could read both files instead of guessing from packages.