Engineering Before Inference: The Question Zero Token Architecture Is Actually Asking

A few weeks ago I heard a developer decline a task for a reason that would have been absurd three years ago: they had run out of tokens.

Not "the build is broken." Not "staging is down." Not even "I'm blocked on a review." The blocker was a usage meter. The work waited on a quota reset — not for lack of skill or will, but because a metered dependency had been wired into the middle of the workflow.

Once I started looking, the pattern was everywhere. The Hacker News thread on Claude Code's weekly rate limits drew over 700 comments, including one user's plain summary: hit the limit and that's it for the week. Cursor spent July 2025 apologizing for a pricing change its users experienced as rationing. There is now a small ecosystem of menu-bar apps whose only job is to watch your remaining tokens the way a fuel gauge watches a tank. And when METR tried to recruit experienced open-source developers for the follow-up to its productivity study — at $50 an hour, down from $150 in the original — enough of them declined rather than code without AI that METR judged its own results unreliable and went back to redesign the experiment; 30–50% of the developers who did take part withheld the tasks they most wanted AI for. TechCrunch put a blunt headline on the pattern in May 2026: coders are refusing to work without AI. The same piece reports Amazon shutting down Kirorank, an internal token-consumption leaderboard, after engineers gamed it by burning tokens, and Uber exhausting its annual AI budget in four months — with its COO saying publicly that the spending produced no measurable increase in projects or productivity.

I want to be precise about what that refusal reports, because it is not a billing problem. The pipeline wasn't down. The editor worked. The compiler worked. The engineer stopped. Somewhere in the last two years, for a meaningful number of developers, the model quietly moved from accelerator to load-bearing — and nobody signed off on that architectural change.

The joke that became a keynote

On March 20, 2026, Kelsey Hightower posted an announcement: "Instead of burning AI tokens, you learn to think for yourself" — introducing what he called the Zero Token architecture. It was a joke. It collected hundreds of reposts, and the best reply on the thread made the counterargument in one line: thinking and completing tasks is exactly what the tokens are being spent on.

Then the joke refused to stay one. Within three weeks Hightower was doing the bit on stage at Nutanix .NEXT, pointing out that a cron job and a shell script already do, at zero token cost, much of what autonomous agents are being sold to do — The Register covered it with an appropriately raised eyebrow. By June 25 it was a main-stage talk at PlatformCon 2026, with a session abstract that dropped the irony almost entirely: "The fundamentals still matter and burning tokens isn't a requirement." Someone registered a parody domain defining the architecture as a shell script with better marketing, established sometime in the 1970s.

Satire does not usually get a conference slot. It got one because it named a discomfort a lot of engineers were already feeling and did not have a term for. And the detail I find most telling: Hightower's stated context was not aesthetic preference but the arrival of organizational token quotas. The joke works because the meter is real.

Underneath the meme there is actual engineering content. Stripped of the delivery, the Zero Token argument says: use AI where the problem is uncertain; once inference has discovered the shape of a solution, export that logic into code or configuration and run it from then on without further inference; do not use agents to paper over infrastructure you should have fixed; and treat the deterministic path as the default for anything stable. That is not an anti-AI position. It is a position about where knowledge should end up living.

But I think "zero tokens" is the wrong banner for it, because it aims the argument at the cheapest thing on the invoice.

Every inference call is an architectural decision

I should establish where I stand, because this essay is not a renunciation. I use AI heavily — agents run scheduled maintenance on my own repositories, drafts of my experiments start in a model's context window, and a good fraction of what I have learned this year I learned faster because I could interrogate a model about it. AI was never meant to replace engineers; its real value is removing repetitive work so more time lands on problems worth a human's attention. I hold that view more strongly, not less, after this research.

Which is exactly why the framing matters. The trend I keep noticing is not developers using AI too much. It is developers delegating a different thing than they think they are delegating — handing over not the typing but the architectural thinking, the system understanding, the judgment about trade-offs — one convenient call at a time, without ever making that decision explicitly.

So here is the principle I would put on the banner instead: every AI call is an architectural decision.

When I add a queue to a system, I accept eventual consistency and operational surface in exchange for decoupling. When I add a cache, I accept an invalidation problem in exchange for latency. These are trades engineers make with their eyes open. An inference call is the same class of decision, with its own bill: I accept non-determinism (a May 2026 Stanford Digital Economy Lab study found identical agentic tasks varying up to 30x in token consumption between runs — the same request is not the same computation twice), no replayability, latency I cannot bound, a dependency with its own failure modes and its own quota — and, most importantly, knowledge that evaporates when the call returns. The reasoning happened outside the system. Nothing in the repository got smarter.

None of that makes the call wrong. It makes the call a decision — one that deserves the same scrutiny as the queue and the cache, and usually gets none, because the marginal call is frictionless and the costs are deferred.

What convinces me this framing is durable rather than contrarian is who else already holds it. Anthropic's own guidance on building agents — published in December 2024, well before the meme — tells builders to find the simplest solution possible and to add agentic complexity only when the task demands it, noting explicitly that agentic systems trade cost and latency for capability. The 12-Factor Agents playbook, distilled from surveying production systems, describes real "AI agents" as mostly deterministic code with model calls placed at the few points that need them. NVIDIA researchers argued in mid-2025 that most invocations in agentic systems should not hit a frontier model at all. The strongest advocates for fewer and cheaper inference calls are the people selling the inference. They are describing good architecture, not austerity.

Engineering before inference, stated plainly: if deterministic software, an existing tool, or a better design solves the problem, that solution wins by default — it is predictable, testable, debuggable by anyone on the team, and it keeps the engineering knowledge inside the system where compound interest works in your favor. The model is for the residue: the genuinely uncertain, the novel, the discovery. And discovery has a harvest step — when inference finds a repeatable pattern, you export it into code you own. The token savings arrive as a consequence of that discipline. They were never the point.

Cheaper tokens will not save the bill

The standard objection arrives on schedule here: token prices are collapsing, so any discipline aimed at reducing calls is premature optimization against a cost that is solving itself.

Per token, the collapse is real and worth stating fairly. a16z's LLMflation analysis measured the price of GPT-3-class capability falling a thousandfold in three years — roughly 10x per year. Sam Altman has cited a 150x per-token price drop between GPT-4 and GPT-4o in about eighteen months. If total spend tracked unit prices, AI line items would be rounding errors by now.

They are not. Ramp's corporate card data — actual payments from thousands of companies, January 2025 through April 2026 — shows token consumption up 1,001% among the heaviest adopters while AI spend across all businesses on its platform rose 497%. Unit prices roughly halved; the bill still grew sixfold. Satya Nadella reached for the right nineteenth-century economist in January 2025: this is Jevons paradox, where efficiency increases consumption faster than it cuts cost. My own reading of the data says it is Jevons plus two accelerants the coal market never had.

First, the frontier trap, which Ethan Ding described in mid-2025: demand does not chase the cheap model, it chases the best model, and frontier pricing has stayed roughly flat while only the trailing tiers deflate. Ramp names the mechanism outright — model tier migration, a change in cost structure that happens without anyone filing a purchase order. Premium reasoning models run 10–20x budget-tier rates, and among Ramp's heaviest adopters the median share of OpenAI and Anthropic spend going to premium models reached 14% by May 2026 and is still climbing. Companies pocket the price decline and immediately trade back up.

Second, consumption per task is exploding. Gartner's March 2026 estimate puts agentic workflows at 5–30x the tokens of a chatbot exchange; the Stanford study measured agentic coding tasks consuming on the order of 1000x more than code chat, largely because agents re-read their accumulated context before every action — re-sent context alone accounted for about 62% of agent inference bills. Usage per developer is not growing linearly with adoption; the workloads themselves are token-hungrier by construction.

And tokens are only the visible line. Cockroach Labs' analysis of agentic AI at scale puts inference at roughly 20% of total cost of ownership once evaluation harnesses, monitoring, guardrails, retrieval pipelines, and governance are counted. Debugging non-determinism is its own budget: one healthcare company watched its inference bill go from $12,000 to $68,000 in six weeks because retrieval faults made agents retry — and every retry re-sends everything. A Deloitte survey found fewer than a third of organizations can attribute their AI spend to measurable outcomes. Gartner now predicts over 40% of agentic AI projects will be canceled by end of 2027, citing escalating costs, unclear business value, and inadequate risk controls.

I read that prediction as an architecture verdict, not a technology verdict. A canceled agentic project is usually a system where inference calls were added because they were easy, not because anyone decided they were the right component — the accumulated result of skipping the same small decision a thousand times.

Token costs may keep falling. Total spending will keep rising anyway, because usage scales faster than unit prices fall. Which means "wait for cheaper tokens" is not a strategy. The only lever that was ever fully in an engineer's hands is the decision about which calls need to exist.

The ledger nobody invoices

Even the total-cost-of-ownership argument undersells the problem, because the largest cost never appears on an invoice at all.

The evidence deserves to be handled carefully, so let me take it at its strongest and its weakest. A CHI 2025 study from Microsoft Research and CMU surveyed 319 knowledge workers and found that higher confidence in the AI correlated with measurably less critical-thinking engagement — the work shifts from reasoning to supervising. METR's mid-2025 randomized trial found experienced open-source developers were 19% slower with AI assistance while believing they had been 20% faster; the perception was not merely optimistic, it was inverted. GitClear's analysis of 623 million code changes shows duplicated code blocks up 81% since 2023 while refactoring — measured as moved code — collapsed from 21% of changes to under 4%: paste is displacing design. An Octopus Deploy report found juniors shipping 55% faster with AI and unable to explain the code they shipped; 73% of surveyed organizations had cut junior hiring within two years. And by one industry estimate reported by TechCrunch, around 44% of enterprise tokens are now spent fixing bugs that AI wrote.

Now the honest counterweight. METR's own February 2026 follow-up does not merely shrink the slowdown, it reverses the sign: returning developers from the original cohort now measure as 18% faster with AI, newly recruited ones 4% faster — both with confidence intervals wide enough to cross zero. METR flags that even those numbers understate AI's benefit, because the developers and the tasks with the most to gain selected themselves out, though it also concedes the sharply lower pay rate did some of that selecting. The original Copilot trial found a 55.8% speedup on a bounded greenfield task. DORA's 2025 research reports 90% adoption and majority-reported productivity gains, and lands on the frame I find most accurate: AI is an amplifier — it magnifies the strengths of disciplined teams and the dysfunctions of struggling ones. Kent Beck's line — "The value of 90% of my skills just dropped to $0" — continues, less quoted, with the remaining 10% multiplying in leverage: the judgment about what to build and what to ask. Simon Willison's "vibe engineering" describes experts using agents aggressively while classical skills — tests, review, specs, planning — become more valuable, because agents iterate against them.

Both columns of this ledger are real, and I do not think they contradict each other. The pattern running through every study is a widening gap between perceived and actual: developers who felt faster and were slower, workers whose confidence in the model displaced their scrutiny of it, juniors whose code passes review but not a "why does this work" question. Skill erosion does not follow from using the tool. It follows from transferring ownership without noticing — from letting verification lapse into admiration.

Martin Fowler's caveat on the compiler analogy is the sharpest tool I found for thinking about this. Yes, LLMs look like the next abstraction layer, and engineers have climbed abstraction layers before without regret; almost nobody misses assembly. But every previous layer was deterministic — you could forget what the compiler did precisely because it did the same thing every time. This layer moves up and sideways into non-determinism at once. You could afford to forget assembly because the compiler is a guarantee. You cannot afford to forget architecture, because the model is a probability. The analogy that justifies the delegation quietly assumes the one property the new layer lacks.

It is also worth remembering what "vibe coding" originally licensed. Karpathy coined it for throwaway weekend projects — "forget that the code even exists" — and said so at the time. The failure mode of 2025 and 2026 was not that he was wrong; it was an industry importing a weekend stance into systems meant to last a decade.

What I keep on my side of the desk

So what does engineering before inference look like in practice? Not a manifesto — a habit, applied at the moment a call is about to become part of a system or part of a workflow. Mine has the shape of a ladder, and I try to climb it from the top:

Does an existing deterministic tool already do this? The rename refactor, the grep, the scheduled job — Hightower's shell-script bit is a joke with a load-bearing core, and reaching past a tool I already trust toward a model is momentum, not judgment. If no tool exists: can I write a small script once, and own it forever at zero marginal cost? If the model has already solved this exact shape for me before: why am I paying for the reasoning again instead of replaying the result — caching, templating, exporting the logic? Only then do the model rungs start, and they start small: a cheap model for the mechanical majority of calls, the frontier model reserved — deliberately, eyes open — for the residue of real uncertainty where it earns its variance.

The rung I skip most often when I'm tired is the harvest. When an agent discovers something for me — a migration recipe, the diagnosis of a flaky test, the incantation for a build system I touch twice a year — the answer is not the deliverable. The asset is: the script, the test, the paragraph of documentation that makes the next occurrence free, fast, deterministic, and mine. Inference that does not leave an artifact behind is a service I rented. Inference that does is engineering. The gap this opens is already visible in finance data: by June 2026 Ramp found 60% of the businesses it tracks running prompt-cache hit rates of 80% or better while 13% sat below 20% — same list prices, wildly different bills, and the difference is entirely whether someone bothered to stop paying twice for the same context.

And some things never go on the ladder at all, because they are not tasks — they are the job. The architecture. The trade-offs and what they cost. The product vision. The user's experience. The final decision, and the accountability for it. I will discuss all of them with a model, usefully and often. I will not delegate them, for the same reason the delegation cannot work even in principle: a model bears no consequences. Ownership is not a sentimental attachment to craft; it is the mechanism that keeps a system explainable by the people responsible for it.

Four questions now sit between me and the call, and they take about five seconds:

• If my quota were zero today, would this task stop? If yes, I am looking at dependency debt, not productivity.
• Am I about to verify this output, or admire it?
• Is this the third time I have asked the model the same shape of question? Then it wants to be a script.
• Will anything in the system be smarter after this call — or only the transcript?

The developer who ran out of tokens was not doing anything foolish; the tools are extraordinary, and the pull toward them is the pull of everything that works. My own meter has hit zero more than once. The difference I am arguing for is small and entirely internal: that the meter's state should never decide whether engineering happens today.

That is the question underneath Hightower's joke, and it is a better question than the joke's critics or its fans usually give it credit for. Not "should you burn tokens" — burn them; some problems deserve a frontier model and a full context window. The question is what remains when they are gone. If the answer is a system you understand, tools you built, judgment you have been exercising rather than outsourcing, then AI has been doing exactly what it was always for: amplifying an engineer. If the answer is that work stops — then somewhere along the way, optimization quietly became abdication, one frictionless call at a time.

A zero-token day should only ever be a slower day. The moment it becomes a zero-progress day, the tokens were never the thing that ran out.