---
title: "Every New Requirement Has a Cost. Make It Visible."
url: https://tiarebalbi.com/en/blog/every-new-requirement-has-a-cost
markdown: https://tiarebalbi.com/en/blog/every-new-requirement-has-a-cost.md
description: "How I added moderated usernames to a live leaderboard with TypeSafe's Jev on Vercel AI Gateway: four costed options, a labelled set and a review queue."
author: "Tiarê Balbi Bonamini"
locale: en
published: 2026-11-10
updated: 2026-11-10
category: "Engineering"
translation: https://tiarebalbi.com/pt-br/blog/every-new-requirement-has-a-cost
---
# Every New Requirement Has a Cost. Make It Visible.

When I added sign-in to Bullpen in [Part 4](https://tiarebalbi.com/en/blog/saga-pattern-time-not-services), I made one choice on purpose: nobody's GitHub name would ever appear on the site. Every player gets a neutral label instead, "Player" plus four characters of a hash, like "Player 6FEF". ADR-0020 records it. It wasn't meant to be the final experience. It kept a sign-in identity from turning into a public one.

Then a requirement arrived that the plan for this series didn't have: players want their own handles on the leaderboard, and those handles need moderation that is strong, smart and efficient. It sounds like a small change to the screen. It isn't. It reopens a decision Part 4 made deliberately, and it puts text chosen by strangers in front of everyone who reads the leaderboard.

I could have said no. The plan was set, and the next part is about the market open. But the answer to a new requirement is never just "no". It is "I understand it, and here is what it costs". A price isn't a promise to build. It is the information needed to decide, and sometimes it says "not now". Then whoever owns the requirement picks. On Bullpen I'm the one asking and the one building, which makes it easy to skip that step, so this post writes the whole exchange down.

This is a bonus between Part 4 and Part 5, outside the six numbered parts. The decision here is a different kind: not "how should this be built", but "should this be built yet, and at what price". The answer I built moderates usernames with TypeSafe's Jev, a decision model, through Vercel AI Gateway, with a review queue behind it. How I got there matters more than the model.

## The requirement as it arrived

The requirement came up on October 16, while sign-in was still being built. Part 4 shipped sign-in on October 26 with the neutral labels and left handles for later. A leaderboard of "Player 6FEF" works, but nobody plays a league to be a hex code.

So the need, written the way I'd want it from someone else:

* **Who:** every signed-in player, and every visitor who reads the leaderboard.
* **What:** a handle each player picks, shown wherever their rank is shown.
* **The condition:** a handle that insults, threatens or impersonates never reaches the page, and a normal handle doesn't wait days for a human.
* **Why now:** sign-in is live, so players exist, and the leaderboard is the first place they'd see each other's names.

"Strong, smart and efficient" isn't something a test can check, so the first job was to turn it into numbers. I fixed these targets on November 1, before the build and before any handle reached the model, the same way Part 3 set its cold-start targets first. No model can promise zero bad handles in production, so the strong target is about a fixed test set, and production misses are counted separately:

| Word | What it means here | Target |
|---|---|---|
| Strong | No handle from a labelled set of must-reject handles is accepted without a human | 0 in the test set; misses in production tracked on their own |
| Smart | Clean handles don't pile up in front of a human | At most 1 in 20 clean handles sent to review |
| Efficient | Fast to answer, cheap to run, little human work | Model call p95 under 500 ms; under $0.0001 a decision; under 10 handles a week in review, and the share of all submissions reviewed |

## Four answers, each with a cost

Each option below is a real answer to the requirement.

**Not now.** Keep the neutral labels until after Part 6. Cost: zero code, zero risk, and a leaderboard of "Player ####" for another month. Handles would land after the series ends. This is the honest default when a requirement shows up mid-plan, and it was the cheapest answer on the table.

**Now, with a blocklist.** A list of banned words, checked in code. Cost: an afternoon, nothing to run, nothing external. It's weak in both directions. It rejects innocent names that happen to contain a banned string, and it accepts anything spelled creatively. Every miss becomes another entry in a list I have to keep.

**Now, with human review only.** Every new handle waits for me. Cost: no new dependency and no false acceptances, but a player who signs up at 9 p.m. waits until I look, and the queue grows with every sign-up. It's strong, but it's neither smart nor efficient, and it puts my time on the critical path.

**Now, with a decision model behind a review queue.** Send each new or changed handle to a model that returns a probability, accept it above a threshold, and queue it for me otherwise. Cost: a new external dependency, a per-call price, a new ADR, a new place where data leaves the system, and time taken from Part 5.

For that last option I looked at Jev, from TypeSafe, which has been on [Vercel AI Gateway since September 16](https://vercel.com/changelog/typesafe-ai-jev-now-available-on-ai-gateway). It doesn't write text; it answers typed questions with probabilities. TypeSafe reports it as "up to 193.6x faster and 444.6x cheaper than LLMs" on its own evaluations, which this post treats as a claim until Bullpen measures it. A general language model asked for structured output would also work, but its answer is generated text shaped into a structure rather than a probability, and its per-token price on the same gateway is higher. The [gateway's decision docs](https://vercel.com/docs/ai-gateway/modalities/decision) can even rerun an unsure decision on a language model automatically. I don't use that. It would bring another provider, another behaviour and another privacy decision, all without me choosing them.

## What Jev costs on Vercel AI Gateway

It's the only option with numbers I had to look up, so here they are, from the documentation and before any build.

| Cost | What the docs say | What it means for Bullpen |
|---|---|---|
| Price | Jev's input costs $0.042 per million tokens on [its AI Gateway page](https://vercel.com/ai-gateway/models/jev), and output is free. The docs' example decision used 275 input tokens and cost $0.00001155 | About $0.12 for 10,000 checks, if a check costs what the example did. The real count depends on the question and its criteria, so the build reads it from the gateway's cost field |
| Free tier | AI Gateway's free credit covers [a subset of models](https://vercel.com/docs/ai-gateway/pricing), with no amount stated; Jev's page shows a price and no free-credit entry | Still unconfirmed. The measurement run went through with no refusal for credit, but a response doesn't say which credit paid for it |
| Data | Jev supports zero data retention and no training as request-level options, but the gateway offers them only on Pro and Enterprise plans | Bullpen is on Hobby, so the handle goes to TypeSafe under TypeSafe's terms. A handle is public by design, so the call sends the handle text and nothing else: no user id, no email, no session |
| Latency | No figure published for Jev on the gateway | Measured below: p95 of 215 ms from my own machine |
| Failure | The gateway answers 429 when a rate limit is hit, and refuses when credit or budget runs out | Covered below: no failure can put an unchecked handle on the page |

TypeSafe's [privacy policy](https://typesafe.ai/privacy) says it doesn't train on input and doesn't share it beyond its service providers, but it states no retention period, and its own zero retention is for enterprise customers. Vercel's [gateway FAQ](https://vercel.com/docs/ai-gateway/faq) says the gateway doesn't keep prompt content. For a handle that is about to be public anyway, that is enough. For anything private, it wouldn't be.

The money is small. The real costs are the dependency, the data leaving Bullpen and the week this takes from Part 5. That last one is what the requirement's owner should weigh, because it is the cost that moves a date. I planned this bonus for November 9, with Part 5 on November 16 and Part 6 on November 23. It went out on November 10 instead, and the two parts after it each moved a week, to November 23 and November 30. The build wasn't what moved them: it took about two hours of work with the agent on November 1. The steps only I could do took nine days: creating a database role, setting a variable, running the measurement with a key. I priced the code and forgot to price my own attention.

## The direction, and why

I chose the fourth option: Jev through AI Gateway, a threshold, a review queue, and a manual path for me.

The blocklist fails "strong" and "smart" at once. Human review alone fails "efficient", and its cost grows with every player who signs up. "Not now" was a fair answer, and I'd have picked it if the model option had come back expensive. It came back as cents, plus a dependency I can contain in one place. The containment matters more than the price, so it gets the most space below.

**What would flip it.** Three things. The labelled set shows the model accepting handles it must reject, which fails "strong". The review queue grows past the target, which fails "efficient". Or TypeSafe's terms rule out sending handles at all; as read on November 1, they don't. Any of these sends Bullpen back to human review until there's a better option, and that path is built anyway.

## The scope, in and out

A requirement that arrives mid-plan tends to grow while it's being built. So the scope was written down before any code, and the build is checked against it:

* **In:** one Boolean question per new or changed handle, "Is this handle acceptable to show publicly?", with written criteria for true and false. Accept only above a threshold; everything else goes to review. One module in the code calls the gateway. A manual path lets me approve or reject queued handles, and it works when the model doesn't.
* **Out:** anything on the money path (orders, prices, buying power); market-data checks; any decision that produces a number; any other AI feature. No model output ever changes an account.

Before any model is asked, plain rules run in code: 3 to 20 characters, letters, digits, underscores and hyphens, unique ignoring case. A handle that breaks one is a form error and costs nothing.

The threshold comes from a labelled set, not from a guess. The set has 146 handles: 106 clean and 40 that must be rejected. I wrote all of them before the model saw any, and none came from a real player. The reject set is coded and mild, with leetspeak and padding rather than slurs spelled out, so the slur rows use placeholders and are a weak test; I'd rather say that than ship a file of slurs in a public repo. The rule for the threshold: the lowest value that approves no reject, with a margin, unless that sends more than 1 in 20 clean handles to review, in which case I choose the trade. Applied to the run, that rule gives 0.90: the highest score any reject got was 0.88, and 0.90 adds a small margin.

## When something goes wrong

The rule for every failure is the same: the player keeps their neutral label until a handle is approved, and the leaderboard never goes down because moderation did.

* **The model times out (2 seconds) or is unavailable:** the handle goes to the review queue.
* **The answer is malformed, missing, or comes from another model:** the handle goes to the review queue.
* **The gateway is rate-limited, the key is missing, or the credit or budget is spent:** the handle goes to the review queue.
* **The probability sits exactly at the threshold:** only above it approves, so the tie goes to a human.
* **A good handle is flagged:** it waits in the queue, and I approve it.
* **A player changes an approved handle:** they go back to "Player ####" until the new one is approved. The old name doesn't stay up while the new one is checked.
* **My review is slow:** the player keeps "Player ####" a little longer. Nothing breaks.
* **The model is confidently wrong:** a bad handle that scores above 0.90 is approved and goes public, and it never reaches the queue, because the queue only sees what the model doubts. This build has no answer for that case yet. It is the costliest failure on this list, so the next change is a way to report an approved handle and for me to take it down.

One more case came out of the code review: every change is a paid call and maybe a queue entry, so one player changing their handle in a loop could spend the budget. A player can now save at most 5 handles in 24 hours, checked before the model is asked. Handles stuck as pending, because a decision couldn't be saved, show up on my review page after 10 minutes instead of waiting for the player to try again.

## What the change touched

A new requirement is also a change to the architecture, so it gets the same impact list as a planned part.

* **Model:** the bonus has its own moment in the CALM timeline, between Part 4 and Part 5. It adds one external system, the AI Gateway, with TypeSafe behind it, and two data nodes: the `handles` schema and a published view of approved handles. The trading app gains three edges: to the gateway over HTTPS, to the `handles` schema as its owner, and to the approved-handles view, read-only.
* **Data:** handles and the review queue live in their own schema, `handles`, owned by a new role, `handles_owner`. Part 3's rule is one owner per transaction, and a handle decision has nothing to do with an order. The trading side can read one thing from it, `handles.approved_v1`, and nothing else; the handles role can't touch `trading` at all. Database tests prove both. The review queue is a table in Postgres, not new infrastructure: it holds a handful of pending names, and what it needs is durability, not throughput. It has nothing in common with the market-open load Part 5 deals with.
* **Leaderboard:** the leaderboard is a materialized view with its own refresh rules ([ADR-0014](https://tiarebalbi.com/en/blog/database-per-service-free-tier-data-ownership)). One migration re-creates it with a join to the approved-handles view, so an approved handle replaces the neutral label at the next refresh. A pending, queued or rejected handle never does. The view refreshes when a read finds it older than 5 minutes, so an approved handle shows within about 5 minutes of a visit. It didn't need a new read model.
* **Decisions:** ADR-0027 records the four options, the choice and what would flip it. ADR-0020 now says handles are no longer deferred.
* **Cost:** a new line in the cost ledger for gateway spend, and an AI Gateway budget of $5 that cuts off spend before it surprises me, with an alert at 75%. At the measured cost, $5 covers about 311,000 decisions, so the cap is there for a bug or an abuser, not for normal use.
* **Later parts:** Part 5's load test now has one more external call in the handle path to account for. Part 6 gets one more decision to grade.

## The rules that keep it inside the scope

A scope written in a post fades. A scope written as rules fails a build. These are four of the new lines in Bullpen's ADL file:

```text
ASSERT(ONLY apps/web/app/lib/moderation READS AI_GATEWAY_API_KEY)
ASSERT(ONLY apps/web/app/lib/moderation READS ai-gateway.vercel.sh)
ASSERT(ONLY trading WRITES trading)
ASSERT(trading READS handles ONLY THROUGH handles.approved_v1)
```

A fifth forbids the trading and market-data code from importing the moderation module at all. Each rule has a fixture that breaks it on purpose, and each fixture fails the check with the file, the line and the rule it crossed. With the bonus in, `check:arch` runs 12 checks over 61 rules, all passing.

The rules caught something during the build, and it was my own instruction. I'd asked for ADR-0027 to be written as Proposed. But ADR-0016, from Part 2, says only an Accepted decision can put a new external system in the model, and the model-currency check failed the moment the gateway appeared. The honest fix wasn't to weaken the check. The decision had in fact been made on November 1, before the build, when I approved the direction and the scope, so ADR-0027 is Accepted and says so. The guardrail made me record what had actually happened.

## What the build measured

I ran the 146 handles once each, from my own machine, one request at a time. The gateway's nearest edge answered from Portland. No call failed. Three of the four targets held, and one didn't:

| Target | Result | Measured |
|---|---|---|
| 0 rejects accepted | Met | 0 of 40 at a threshold of 0.90 |
| At most 1 in 20 clean handles to review | Missed | 17 of 106, about 1 in 6 |
| p95 under 500 ms | Met | p50 156 ms, p95 215 ms, slowest 311 ms |
| Under $0.0001 a decision | Met | $0.00001608, from the gateway's own cost field; $0.0023 for all 146 |

The cost came in about 39% above the docs' example and still six times under the target. The latency numbers describe that one run, sequential, from my machine. They aren't what a player waits for: a Vercel function will measure its own number, and the page adds its own time on top.

One caveat applies to every row. The same 146 handles chose the threshold and then reported how well it did, so this is a development set, not proof of how the model handles real players. A second set, labelled separately and never used to pick the threshold, would be the honest test. So would the real submissions, which Part 6 reports.

The miss is the interesting part, and the scores show where it comes from. Clean handles scored between 0.78 and 0.97, with a median of 0.94. Rejects mostly scored low, with a median of 0.25. But 4 of the 40 scored above 0.8, and the highest reached 0.88. The two groups overlap, so no threshold meets both targets on this set:

| Threshold | Rejects accepted | Clean handles to review |
|---|---|---|
| 0.80 | 4 | 2 |
| 0.85 | 2 | 4 |
| 0.90 | 0 | 17 |
| 0.95 | 0 | 87 |

At 0.85 the queue meets its target, and two handles I wrote to be rejected reach the page with nobody looking. At 0.90 nothing bad gets through, and about one clean handle in six waits for me. The clean set is full of trading jokes, like a stop-loss pun, a gold digger and a bet on red, so the model's hesitation isn't absurd. But it's a miss against a target I set, and the post would be worth less if I moved the target after seeing the result.

So I'm keeping 0.90. A wrongly accepted handle is public and costs trust; a wrongly queued one costs a player a short wait and me a click. That's my call to make, as the one who owns the requirement, and ADR-0027 records it with the numbers. The flip condition still stands: if the real queue grows past 10 handles a week, the question's wording gets rewritten and re-measured, or the decision changes.

The comparison with a blocklist shows that list's weakness more than the model's strength. A plain substring list would block `Espresso_Eddie`, because it contains "die", and `Bass_Line`, because it contains "ass". It can't read context. The model approved both. The reverse also happened: `LunaLovegood` and `Pork_Belly` contain nothing a word list would catch, and the model sent both to review, at 0.90 and 0.87, so there the model is the stricter of the two. What this run doesn't show is the half that matters most: whether the model catches harmful handles a blocklist would miss. The printed results don't have each reject's score, so I can't claim it.

The first week in production adds the two numbers a test set can't give: the share of real submissions that went to review, and the queue size. I'll report them in Part 6, next to the other decisions it grades.

## Make the cost visible

The useful habit isn't saying yes to new requirements. It's refusing to answer before the price is on the table. A price isn't an acceptance. Sometimes it justifies a no, sometimes a "not now", and sometimes, as here, it shows the requirement was cheaper than it looked in money and dearer in two things I hadn't priced: one handle in six waiting for a person, and nine days of my own steps. If the model option had cost a month instead of a week, I would have taken "not now".

What made "now" safe wasn't the model. It was writing the scope down before building, then turning it into rules, so the requirement could arrive mid-plan without taking the plan with it. When the next one arrives in your system, ask what it costs before you ask whether to do it, and settle who gets to decide.