The Jev AI Model Can't Hallucinate. It Still Got 37% of Emails Wrong.

​TechCrunch's launch-week story on TypeSafe's Jev AI model put the pitch in one line: "because users define the outputs in advance, it cannot hallucinate." The day before, an independent benchmark asked Jev a single question about 2,000 emails, whether each one was phishing, and it got 37.4% of them wrong. Both statements are true.

Jev guarantees the shape of an answer, not its truth. It is a fast, cheap decision function for small, bounded questions that I can label and threshold. It is not a smarter language model, and it is not a chatbot that happens to be quick. Since TypeSafe AI opened early access on 15 September, I have read the launch material, the docs, the first independent tests, and the source of the Pydantic AI adapter, which I installed and ran. What follows is where the launch claims hold and where they stop.

What the Jev AI model was built for

TypeSafe calls Jev a System One model, borrowing Kahneman's split between fast, intuitive judgment and slow deliberation. The launch post describes it as a model that takes unstructured state and returns typed decisions with probabilities. It is not built to return prose. There are three kinds of question:

• Noul: a yes/no question, answered with the probability of yes.
• Choice: pick one of up to 255 options, with a probability for each.
• Score: place the input on a rubric of up to 10 levels.

Because the answer space is fixed in advance, Jev does not generate token by token. It answers every question in a request in one parallel pass. That is where the speed comes from: 70 to 500 ms end to end by TypeSafe's numbers, at $0.042 per million input tokens, with output tokens free.

The intended jobs are the ones a program needs, not the ones a person asks for. TypeSafe lists routing, classification, scoring, smart conditionals inside workflows, map-reduce labelling over large datasets, and checking or guardrailing the output of other models. The same launch post concedes that language models remain the better tool for chat, copilots, and coding agents. The builders did not claim it could do everything. Parts of the coverage leaned that way: a write-up on the launch hype records a newsletter calling it a "ChatGPT moment for decisions", and in the discussions I have followed since, that has stretched into treating Jev as a faster drop-in for a language model.

Adoption has been fast. Vercel reported that nearly 13% of paid AI Gateway teams were calling Jev within 24 hours, more than twice any previous launch. That is why the claims below are worth checking now, before a wave of integrations bakes them in.

Four launch-week claims against the evidence

Cannot hallucinate means the answer parses, not that it is right

This is true in a narrow sense that matters. Jev cannot return a value outside the schema. The Pydantic AI adapter enforces that before a request leaves the machine. Here is a ticket-triage output type in Python, with one field typed as a free string:

from pydantic import BaseModel, Field
from pydantic_ai import Agent
from pydantic_ai.exceptions import UserError


class Triage(BaseModel):
    """How to handle an inbound support ticket."""

    queue: str = Field(description="Which team should own this ticket?")
    urgent: bool = Field(description="Does it need a reply within an hour?")


agent = Agent("typesafe:jev-latest", output_type=Triage)

try:
    agent.run_sync("I was billed twice for invoice 4411.")
except UserError as err:
    print(str(err).split(". ")[0])

pip install "pydantic-ai-slim[typesafe]"
PYDANTIC_AI_NO_BANNER=1 TYPESAFE_API_KEY=unused python triage.py

On pydantic-ai-slim 2.48.0 this prints Output field 'queue' is not supported by this model. The fake API key never reaches TypeSafe, because the adapter refuses the schema first. The fix is to declare the answer space: queue: Literal["billing", "bugs", "sales"]. In my own run, the adapter also refused, before sending anything, a plain str output, an int field, and a field with 256 options. A valid schema went through to the server and came back with a 401.

So the no-hallucination guarantee means the answer will always parse, and I value that. It does not mean the answer is right. In the jev-phishing-bench run on jev-1.13.0, asking whether each email was phishing scored 62.6% accuracy against 81.3% for Claude Haiku 4.5. Jev caught 43.2% of the phishing emails; Haiku caught 76.4%. The benchmark author's summary is blunt: "Jev's own verdict loses clearly on accuracy." A wrong answer picked from a valid set is harder to notice than a malformed one, because nothing downstream complains.

Bias gets the same protection. Simon Willison asked Jev to score Bay Area cities on "Good city?" and it ranked Cupertino first and East Palo Alto last. Every answer was well-typed. The prejudice was inside the probability, where no schema can see it.

193.6 times faster, against frontier models on TypeSafe's own workflows

The launch post says the 193.6x speed and 444.6x cost multipliers come from its workflow evals, and that TypeSafe expects them to sit at the high end of real-world gains. In that table, the other models' workflow runs took 10.1 to 86.5 seconds per case, against Jev's 0.4 seconds at $0.0004 per case. Claude Sonnet 5 tied Jev at 67.8% and took 78.1 seconds, about 195 times Jev's 0.4, close to the headline speed ratio. That 67.8% is agreement with reference labels made by averaging GPT-6 Astra and Claude Fable 5.1, not agreement with ground truth, and the launch post concedes the choice biases results toward those two vendors' models.

The multiplier depends on what sits on the other side. Claude Haiku 4.5 appears in the same table at 12.5 seconds and $0.0195 per workflow case, about 31 times slower and 49 times more expensive than Jev. In the single-question phishing benchmark the gap shrinks again: Jev's median latency was 239 ms against Haiku's 687 ms, about 2.9 times faster, and it cost $0.038 per 1,000 emails against $0.462, about 12 times cheaper. Network matters at this scale. From where that benchmark ran, 163 ms of Jev's 239 ms was network floor, against 18 ms for Haiku. Take the floors out and the model-side gap is roughly 9 times, not 2.9.

Calibrated probabilities still need a threshold tuned on labelled data

TypeSafe trains Jev with a method it calls Reinforcement Learning for Calibrated Decisions, and the probabilities are the most useful thing it returns. They still need checking before use. On the phishing single verdict, the benchmark measured an expected calibration error of 0.154 for Jev against 0.097 for Haiku, so Jev's stated probabilities sat further from the observed hit rates than Haiku's on that data. TypeSafe's own jaggedness notes for jev-1.13 add that the probability of yes and the probability of no are not guaranteed to sum to one. They also say a Choice answer and a Noul answer about the same fact can disagree, so a threshold tuned on one primitive does not transfer to the other.

The adapter adds a trap of its own. A bool field rounds the probability at a default threshold of 0.5. When I called the adapter's rounding function directly, 0.51 and 0.99 both came back as True. From the adapter source, the margin survives only as a "confidence" in the response's provider details: 0.02 for the first, 0.98 for the second. If the probability is what I will act on, I declare the field as a float bounded between 0 and 1, which the source passes through unrounded.

The adapter's docstring also says a versioned model id, such as jev-1.13.0, is what to use once a confidence threshold has been tuned against it. Thresholds belong to a model version, not to the model family.

What it cannot do, according to its own docs

The docs are the best rebuttal to the drop-in framing, because the vendor wrote them. The jaggedness page opens its list with a line I would print above any integration: Jev "answers the question you wrote, not the one you meant." The rest of the list covers counting and arithmetic, hex values and RGB triples, date ordering and durations, double negatives and multi-hop reasoning, long state full of irrelevant detail, injected instructions it is not hardened against, contradictory criteria, and text generation, which it was not trained for and is slow at when forced.

Sean Goedecke adds the structural limit. A model that answers in one pass cannot spend test-time reasoning, so its judgment tops out around a non-reasoning language model's. The adapter's model profile makes the rest concrete: 64,000 tokens of context in total, of which the state plus the longest question must fit in 32,000. It has no text output, and it accepts no images or files.

Where it earns a place in the pipeline

The most useful result in the phishing benchmark is not the 62.6%. The same author then asked Jev five narrow questions about each email instead of one broad one: is there a domain mismatch, a link on free hosting, a lure, urgency pressure, a generic sender. A cross-validated logistic regression over those five probabilities, fitted on the benchmark's labels, reached 95.1% accuracy, with an AUROC of 0.988 and a calibration error of 0.027. The free-hosting signal alone, used as a fixed rule, reached 89.5%.

The diagram below shows the two shapes side by side. Look at where the composition happens: in the second shape, Jev only answers atomic questions, and the verdict is assembled in code.

That is the shape I would design around. Jev is a branch condition inside a deterministic flow, the pattern I argued for in The Deterministic Backbone. Each answer lands in one of three bands. Above a tuned threshold, the code acts. In the uncertain middle, it hands off to a language model or a person. Below the lower bar, it takes the default path. The adapter already supports the hand-off. When Jev picks a tool whose arguments it cannot fill, the adapter raises ToolCallProposed, and a FallbackModel that puts a language model behind Jev can take the call.

At $0.042 per million input tokens, a 500-token judgment costs about two thousandths of a cent. That changes where a judgment can go: into places that used to hold an if on a regex. It is the argument from Engineering Before Inference, turned around. Every AI call is still an architectural decision. The cost has moved from the invoice to the accuracy I now have to measure.

When to use Jev:

• Routing a ticket, message, or request to one of a fixed set of queues, agents, or tools.
• Deciding whether a workflow should continue, retry, ask the user, or stop, one of the uses Vercel names in its launch note.
• Guardrails and checks on another model's output: a yes/no on a specific, written criterion.
• Triage and prioritization, with the probability kept as the score.
• Reranking search results or candidates by a stated relevance question.
• Labelling millions of rows where a language model's price or latency would rule the job out.

When not to use Jev:

• Anything whose output is prose, code, or a summary. That is a language model's job.
• Arithmetic, counting, and date logic. I compute those in code and pass Jev the result.
• Input an attacker controls, when the decision grants access or moves money. The permission check stays in code.
• Decisions about people, such as hiring, credit, or moderation of individuals. Opaque scores hide bias, as the city test showed.
• Multi-step reasoning, or a question whose answer space I cannot list in advance.
• State that needs more than 32,000 tokens of mostly irrelevant context. I filter first.

What I would measure before it acts alone

If I wired Jev into a pipeline tomorrow, I would run it in shadow mode first and measure five things before letting a single answer take effect.

• Accuracy on my own labelled decisions, not on the vendor's evals. A thousand decisions of 500 tokens each cost about two cents to replay.
• Calibration per band. Of the answers that came back at 0.9 or higher, what fraction were right? The threshold comes from that number, not from 0.5.
• One broad question against its decomposition. The phishing gap was 32.5 points, and I would expect a gap of that kind wherever one verdict is really a combination of separate facts.
• The decision step's share of end-to-end latency. If it is 20% of a request, an instant decision makes the whole request at most 1.25 times faster. That is the only speedup users will notice.
• Disagreements with the language model already in the loop, read by a person. Disagreements are where the jaggedness list turns into specific cases.

I would also pin jev-1.13.0 rather than jev-latest the moment a threshold was tuned, and re-run the shadow set on every version bump. That is the same discipline the TimesFM 2.5 run taught me: a model that beats a baseline on steady data can lose to a one-line baseline after one realistic change. The type system tells me every answer will parse. Only the labelled log tells me whether it is right.