Writing

All posts

Long-form notes on software, distributed systems, and the craft of building. Shipping one a week.

All AI Backend Distributed Systems Engineering Research

20 posts

May 28, 2026Distributed Systems

Drop the Right Requests First: Priority-Aware Load Shedding Under Overload

Static RPS caps shed the wrong traffic. Concurrency is what saturates a service, not request rate. From my notes after reading the InfoQ piece on overload protection, Uber's January writeup on Cinnamon, and Netflix's QCon SF talk on service-level prioritized load shedding, here is why latency is the right control signal — and how a small priority taxonomy plus an adaptive concurrency limit keep the cheapest traffic shedding first.

Read →

May 26, 2026Distributed Systems

Actor-per-Entity vs Postgres Optimistic Locking: A Seat-Reservation Bake-off

I ran the same hot-key seat reservation workload two ways: Postgres with a version column and retries, and a single actor per seat. The actor design did not scale better — it moved the hard problem from concurrency control to routing and rebalance correctness, and that trade was the easier one to reason about under hot keys.

Read →

May 23, 2026Distributed Systems

Auditing a Scala Service Against Chad Fowler's Four Regenerative Constraints

I walked a Scala order-processing service from my notes through Chad Fowler's four regenerative constraints. Two passed for free, two would force a real redesign. Here is what I learned about where "loosely coupled module" ends and "regenerative component" begins, and which parts of the redesign I would actually pay for.

Read →

May 21, 2026Research

Preregistering Experiment #1: How Fragile Are Production JSON Extraction Prompts?

The first concrete experiment in the precision research line — committed in public before the data is collected. Five perturbation classes, fifteen variants, four metrics, four falsifiable hypotheses, and a runnable companion repo with tests.

Read →

May 19, 2026Backend

Durable Execution Isn't About Agents — It's About Replayable Backend Workflows

I came to durable-execution runtimes through the agent press, but the constraint that surprises everyone is determinism on replay. These are my notes from working a six-step payment reconciliation as a Restate workflow in TypeScript — the line that broke replay, the mental model that fixed it, and the trade-offs that come with the pattern.

Read →

May 16, 2026Research

AI Prompts: How Good and How Bad They Are — Opening a New Line of Research

An honest look at where prompts work, where they quietly fail, and the assumption we stopped questioning — that AI must make mistakes. The opening shot of a research line on moving from "best effort" to specifiable, measurable precision.

Read →

May 14, 2026AI

Code Graphs for Coding Agents: The Delivery Shape Matters More Than the Algorithm

I spent a weekend pointing a coding agent at a 480k-line Go monorepo and watching it grep-loop through 38 tool calls on one question. AST-derived code graphs fix that, but the delivery shape — local stdio MCP, remote service, or skill — changes the economics more than the graph algorithm does. Here is where I would put one in 2026, with a minimal Go indexer I can drop next to the agent.

Read →

May 12, 2026Distributed Systems

AckWait Is a Contract: How a 30-Second Default Took Down My JetStream Consumer

I lost an evening to a NATS JetStream pull consumer that doubled its work in production. The cause was three lines of ConsumerConfig I never wrote. These are my notes on what AckWait actually counts, why MaxDeliver = -1 is the silent footgun, and the 70-line Go contract I now ship on every JetStream consumer.

Read →

May 9, 2026Engineering

Idempotency Is a Protocol, Not a Key

The first time I shipped idempotency as a UUID header and a Redis lookup, a duplicate charge slipped through a week later. These are my notes on treating idempotency as a four-part protocol — dedup, determinism, concurrent safety, downstream propagation — with a minimal Kotlin plus Postgres implementation that holds up under retry.

Read →

May 7, 2026Engineering

What `dbos ontime` Actually Asks: Building a Distributed Cron on etcd Leases in Go

A 0-click query for `dbos ontime` showed up in my Search Console last week. The reader is not asking about DBOS — they are asking how to run a job every minute, exactly once, across a fleet. From my own notes, an etcd lease, the `concurrency.Election` package, and a fencing token cover that case in under 100 lines of Go, without pulling in a workflow engine.

Read →

1 / 2Next