← Back to Blog
For: AI Engineers, ML Engineers, Platform Engineers, AI Security Experts

Where Jev Belongs in an Agent Harness (Not a Model Swap)

Its ordering is trustworthy and its confidence numbers are not, and that difference decides where in the harness it can safely sit.

ai-engineeringagentic-ai
#jev#system-one-models#calibration#agent-harness#confidence-thresholds#decision-models

In April 2020, the University of Michigan switched off its sepsis alerts.

The model had not changed. Nobody shipped new weights or touched the alerting threshold, which sat where it had always sat, at 6. The patients changed. COVID-19 arrived, elective admissions stopped, and the population underneath the score shifted. Researchers measured the result across 24 hospitals: in the three weeks after each system's first COVID case, the share of patients triggering an alert each day went from 953 of 10,159 to 1,363 of 6,634. Census fell 35% while alerts per day rose 43%, which is what a fixed cutoff does when the denominator collapses under it. Michigan paused the alerts within four weeks of its first COVID hospitalisation, after nurses reported being buried.

A fixed threshold is a promise about prevalence. When prevalence moves, the promise breaks quietly, and nothing in the model reports the breakage.

TypeSafe AI shipped Jev on 15 September 2026, and the integration guides landed inside 72 hours, most of them ending at the same place: pick a confidence threshold and act above it. That is why the opening story is six years old and about a hospital.

Why placement is the decision, not adoption

Jev is the first of what TypeSafe calls System One Models. It does not generate text at all: you send it program state and a set of typed questions, and it returns typed answers with probabilities attached, in a single forward pass, in roughly 70 to 500 milliseconds. It is the decision layer separated from the execution layer, collapsed into one model call. Input costs $0.042 per million tokens, and output is free.

Whether to adopt it is the easy question. Where to put it is the hard one, and that turns on a property most of the launch-week coverage skipped.

Almost every position in an agent harness needs one of two things from a decision model. Some need the ordering to be right: routing a request to the cheaper model, pruning a menu of candidate actions down to the plausible ones, ranking which alert a human should look at first. Nothing in those positions cares what number sits next to the winner. Others need the magnitude to be right: any gated execution of the form if p >= 0.85: act. There the number is the decision boundary, and an ordering that is perfect in rank order will still put that boundary in the wrong place.

A third class consumes neither, and it is the hardest of the three. Code of the form if top1 - top2 < 0.1: escalate, or a relative prune that keeps everything within a tenth of the leader, consumes differences. A per-score calibration map does not repair a margin. Platt rescaling stretches the score axis unevenly, so it changes margins non-uniformly, and isotonic regression is piecewise-constant and can flatten a margin to exactly zero. If your code reads a difference, fit on the difference, not on the score.

Within a single question, Jev's ordering is good on arrival. Across calls it is not established, because every published area-under-curve figure is a within-question measurement. Its magnitude is not good on arrival either, and cannot be made so by anyone but you, because the correct scale depends on the base rate of your queue. TypeSafe has foreclosed the alternative deliberately: the same weights serve every account, with no fine-tuning and no per-customer adaptation, so a deployer-side map is the only mechanism the architecture leaves open. This is the same shape as the argument that you cannot train your way out of hallucination and need an auditable boundary outside the model instead. You cannot train your way to a correct threshold either. So the first question in a design review is not how dangerous the action is. It is which property the code is actually consuming.

TypeSafe's own guidance grades thresholds by consequence: their documented pattern gates a read-only balance check permissively and approve_transfer at confidence > 0.9. That is not wrong, and it is not what this article is arguing against. Consequence is precisely what tells you where to put a boundary, because the expected-cost optimum sits at p* = cost / (cost + gain) and that cost ratio is the consequence of being wrong. What consequence cannot tell you is whether the number you are putting the boundary on has a scale at all. That is the earlier question, and it is the one the guides skip.

The two steps are sequential and both are necessary. Property first: is this number admissible as a probability? Consequence second: given that it is, where does the cost ratio put the line? Run them in that order and the familiar advice still works. Run them in the other order and you are tuning a dial with no markings.

Which is why raising an unfitted threshold is not the fix it feels like. It does buy you something real: on a well-ranked score, moving the cutoff up weakly increases precision and weakly decreases recall, and that is a property of the ranking, not of the calibration. What it does not buy is knowledge. You have traded recall you never measured for precision you cannot state, moved the operating point by an unknown distance along an axis with no scale, and ended up more conservative by an unknown amount with the same inability to report an error rate.

What the launch-week calibration studies already established

The mechanism underneath this is old and the Jev-specific measurements are days old. Both belong to other people, and the article is worth less if I pretend otherwise.

The statistics date to 1973, when Allan Murphy's vector partition of the probability score separated a forecast's reliability from its resolution; DeGroot and Fienberg formalised calibration and refinement as distinct properties in 1983, and Bröcker connected the two literatures in 2009. Guo et al. made it common knowledge in machine learning engineering in 2017, with the result that temperature scaling fixes calibration without changing the argmax. A model can sort beautifully and lie about magnitude. That is chapter one, not a finding.

Three people measured it on Jev before I did. Samuel Sacco's evidence ledger, updated 18 September, re-derives every published Jev calibration figure against a simulated noise floor for that sample size, and concludes that you should "treat Jev's output as a monotone score, not a probability, and fit your own calibration map on a few hundred labelled cases." Adil Muhammad Pervez ran 8,000 judgments against human annotations a day later and put the framing plainly: "A model that emits probabilities has two independent ways to be good, and they fail separately."

Retriever AI got there by building rather than arguing: on 16 September they accepted Jev's highest-ranked browser action with no cutoff at all, and wrote that a cutoff could reject useful actions as easily as mistakes, so any threshold should come from measured outcomes.

What none of them said is the part I want to defend. Sacco gives a global recommendation about the model; Retriever AI made a local engineering choice. What none of the three does is turn it into a rule about positions: that a harness partitions into two classes by which property of the number is load-bearing, and that this partition, not the blast radius of the action, is the first cut you make.

What Jev actually returns

Three primitives. Choice picks one option from up to 255 and returns the winner, the full distribution, and a confidence score. Score places the state on a developer-defined scale of up to 10 levels. Noul returns the probability of a yes/no statement as a bare float.

That last one has a detail worth knowing before you write the integration: a Noul carries no separate confidence field. Langfuse flagged the consequence during launch week: code that reads answer.confidence uniformly across a batch of questions breaks the moment a binary lands in it.

Several other constraints shape where the model can sit at all:

  • Questions batched into one call run in parallel and cannot read each other's answers, so a question that depends on a prior result needs a second call.
  • The model never sees your field names. A question keyed safe_to_publish communicates nothing; only the question text and the option descriptions instruct.
  • With no rationale trace, a correct answer and a wrong one are identical from outside, which makes logging the full distribution the only forensic record you will have.
  • It cannot abstain unless you supply an explicit escape option, so restraint has to be built into your option list rather than expected from the model. MindStudio asked it when the cafeteria closed with only billing, technical and sales available; it answered sales at confidence 0.31. Given an "other" option, it chose correctly.
  • Context is two budgets, not one: 64k tokens per request, 32k for state plus the longest question. TypeSafe's own jaggedness page also documents accuracy degrading as state fills with irrelevant content, so it behaves as a slope before it becomes a wall. Reports of a single hard 32K cliff oversimplify this, and OpenRouter's listing disagrees with TypeSafe's docs. Verify before you design around it.

The wrong way: a threshold nobody in your system fitted

Here is the shape every launch-week guide converges on.

python
from langchain_typesafe import Choice, TypeSafeClassifierclassifier = TypeSafeClassifier()ACTIONS = {    "refund": "Issue a full refund to the customer's original payment method.",    "reply": "Send a written reply. No money moves.",    "escalate": "Hand the ticket to a human agent.",}def handle_ticket(ticket_state: str) -> str:    answer = classifier.invoke(        state=ticket_state,        questions={            "action": Choice(                instructions="What should we do with this support ticket?",                options=ACTIONS,            )        },    )    decision = answer.choices["action"]    if decision.confidence >= 0.85:        return execute(decision.choice)      # execute() is yours; may move money    return execute("escalate")

That code runs, and the types hold. The model cannot return a value outside ACTIONS, which is the guarantee TypeSafe actually offers, and the reason "Jev cannot hallucinate" is a claim about the interface rather than about truth.

The defect is the constant: that 0.85 was fitted on nobody's data. It is a number that propagated from a worked example into published builds inside a week, and it is now the boundary at which this function moves money.

To be fair to the source: the guide that popularised 0.85 says immediately afterwards that it is a starting point rather than a rule, and to set it from your own labelled examples. TypeSafe's docs say the same thing, in almost the same words, and nobody gave bad advice here. What went wrong is that the caveat sits in prose while the number sits in code, and code is what gets copied. LangChain's AutoModeMiddleware(tools=["bash"]) ships the same pattern as a one-liner that blocks risky tool calls before execution, with no threshold parameter exposed and no documented default. It is an excellent piece of engineering sitting in the class that needs a local fit, with no surface on which to perform one.

Threshold Transplant

Threshold Transplant is a threshold that has crossed a base-rate boundary: from the vendor's evaluation distribution, from a blog post, from another queue in your own system, or from your own queue as it was before the mix moved. The ordering survives the crossing intact. The number does not.

Drift is the model changing underneath a threshold. A transplant is the population changing. The boundary can sit between two systems or between two months, and what defines it is that prevalence moved while the number did not. Either way it looks completely healthy while being wrong, because the ranking underneath it is genuinely good.

The mechanism is not new and does not need to pretend to be. Statisticians call it prior probability shift or label shift; clinical prediction calls the symptom poor calibration-in-the-large on external validation. What is new is how cheap it has become to commit, now that a typed decision model hands back a float that looks ready to threshold.

The Epic Sepsis Model is the clean case: the same threshold, the same model, a population that moved underneath it. Three weeks from prevalence shift to nurses drowning.

That example carries a second defect worth stating, because it cuts against the framing here rather than for it. Epic's discrimination was independently poor: external validation reported an area under the curve of about 0.63, against the 0.76 to 0.83 Epic cited. So the sepsis model was not a well-ordered score whose scale broke. It was a mediocre score whose scale also broke. The case survives because the threshold was a promise about prevalence regardless of the ordering underneath it, and that promise broke on its own schedule. Good ordering buys you no protection here. It only means you notice later.

TypeSafe has already conceded a narrow version of this. Their jaggedness page warns not to carry a threshold tuned on a Noul over to a Choice, and publishes its own counterexample. "Is the customer asking for a refund?" puts 0.22 on yes as a Noul; the same question as a two-option Choice puts 0.99 on no. Those are not a probability and its complement, because they are different primitives answering the same question and disagreeing about it. Separately, TypeSafe reports a Noul and its actual negation summing to 1.19 rather than 1.0. Threshold portability is already known to be broken across primitives. It is broken across base rates too, and that case is not in the docs.

The evidence, as of 2026-09-20, against jev-1.13.0

Dated section. Every number below was measured between 16 and 19 September 2026 against jev-1.13.0, the only version that exists, and was checked against its source on 20 September. TypeSafe publishes no calibration guarantee, no recalibration endpoint, and no prevalence-dependent threshold guidance; calibration does not appear on their own known-defects page. Their docs warn that jev-latest moves silently and that thresholds tuned against a version should pin that version. If TypeSafe ships a recalibration, the numbers here expire and the placement rule does not.

Four independent studies now measure Jev's calibration, and they agree that the ranking is strong and disagree, sharply, about everything else.

Pervez's 8,000 judgments against human annotator panels on Civil Comments produced an AUC between 0.903 and 0.912, with raw expected calibration error between 0.157 and 0.209, while rank correlation between stated confidence and the actual flag rate came to 0.96. That is the split in two numbers on one dataset: an area under the curve above 0.90 against an expected calibration error above 0.15. The sorting is excellent and the scale is not, and no single score reports both. At roughly 75% stated confidence, human raters flagged 10%.

The sharpest single result is on the hard sample: asked about insults at the default 0.5 threshold, Jev scored 61.0% accuracy, where answering "no" to everything scores 67.8% (the insult question's own base rate in that sample is about 32%, above the 19.2% pooled figure). That is worse than a constant. That same question ranks at AUC 0.83, so the signal is there. The boundary is in the wrong place, and moving the boundary is free while the signal is not.

His base-rate finding is the one that matters for placement, and it surprised him: the natural sample, at a 2.9% base rate, calibrated worse than the hard sample at 19.2%. Error grew as positives got rarer, and rare-positive queues are exactly where teams most want automation, and exactly where the shipped number is least trustworthy.

A fitted map removed most of that error without touching the ranking at all: AUC moved 0.918 to 0.918. Two things qualify that figure. His repository fits both a Platt sigmoid and an isotonic regression and reports one number, so "a two-parameter fit" is not established. And he ships jevcal, a tool that performs exactly the correction his audit recommends, which is a real interest to disclose even though the code and data are public and the whole run cost five cents.

Now the disagreement. Sacco's controlled 800-item difficulty gradient found compression toward the middle, with Jev understating high probabilities, so a 0.9 threshold gave a 1.000 hit rate at 21.5% coverage, which is conservative rather than dangerous. Sit with that, because it is the strongest objection to everything above: in Sacco's regime an unfitted 0.9 threshold was perfectly safe. That is the point rather than a problem for it. He could say so because he had labels. Without them you cannot tell his world from the next one. The phishing benchmark found the opposite on 2,000 emails: overconfident in every bin, worst at 0.85 to 0.95, where a stated 0.90 bought a 60% hit rate. Its author concluded there is no threshold at which Jev's confidence is safe to route on. Pervez found a systematic lean toward yes.

Then scienthoon found that the sign depends on which primitive you ask: a temperature fitted per question type gives Choice 3.29 and Score 3.40, both overconfident. Noul, on the same 900 inputs, comes back at 0.66, underconfident. Quoting the pooled 2.74 hides the flip.

Sacco says it in one line: "The sign of the error, not the size of the ECE, determines whether thresholding is safe - and a scalar ECE does not carry the sign."

Read that operationally. Four competent teams measured the same model in the same week and could not agree on which direction it is wrong. You cannot look up your correction or infer it from a published ECE; the only way to learn the sign is on labelled outcomes from your own queue. That is precisely what the ordering-versus-magnitude split buys you: it tells you which positions are worth the trouble.

scienthoon's design is the best-conceived experiment in the set: they built a task whose label depends on an organisational policy absent from the text, so no zero-shot model can recover it. Jev scored 44.7% against a 25% chance baseline, which is well above chance and means some signal leaked through surface correlates rather than the hidden rule. It still placed 0.74 average probability on its chosen level. Partial knowledge held with near-certainty is the failure mode, and it is the harder one to catch. On the question of whether the model knows when it does not know, that is a no.

One more finding, unrelated to sign. On one public benchmark, 1,051 of 2,000 option probabilities came back as exactly 0.00. That benchmark is four-option, so this is not the harmless sparsity you would expect across 255 options, and it happened where Jev is otherwise well calibrated. Whether the model assigns a true zero or the response format rounds a small number down to one is still open at the source. Either answer costs you something: temperature scaling cannot repair a true zero, and a rounded one means you cannot recover a full distribution for forensics.

Detecting a transplant without labelling anything

Fitting a calibration map needs labelled outcomes, which is the expensive part and the reason teams skip the whole exercise. Detecting that you need one is nearly free.

Run the gate in shadow mode over a week of traffic and record how often it would have fired, then ask the system of record how often the thing was actually true: how many refunds finance issued, how many flagged tickets a reviewer confirmed. Compare the two rates. Shadow mode is not optional here. Once a gate is acting, it changes what reaches the system of record, and a confirmed-flag rate measured downstream of your own gate is conditioned on that gate and cannot recover the underlying rate.

Pervez's wording test is the reference case. Across four phrasings of the same question against the same state, the area under the curve barely moved, from 0.881 to 0.913. The predicted positive rate did not hold still at all: against a true base rate of 4.8%, the strictest wording he tried still predicted 26%.

A gate firing five times more often than the event occurs is the cheapest signal you will get that something needs investigating. It costs one query against a system you already run. Under-firing is the same signal with the sign flipped, and it is what you will see if your deployment lands in Sacco's regime rather than the phishing benchmark's.

Disagreement does not prove the threshold is wrong, and the failure modes are worth knowing before you act on it:

  • The gate is meant to over-fire. A screening or triage instrument is built for recall, and a predicted positive rate well above the base rate is its specification, not its defect.
  • The two numbers answer different questions. refund_gate asks whether the policy is satisfied. Finance's issued-refund count reflects policy plus goodwill, fraud denial and chargeback rules. Different estimand, no conclusion.
  • The gate sees a filtered population. Anything downstream of a router sees an enriched distribution, so its rate should exceed the system-wide one. Comparing it against a global figure is itself a base-rate error.
  • The sample is too small to divide. At the rare base rates this article says matter most, a week of moderate traffic yields a handful of events. Wait for at least 30 confirmed positives before the ratio means anything.

There is also a partial exception worth knowing, because a reader who knows the literature will raise it. If the only thing that moved is the base rate, and the classes still look the same, prior-shift correction recovers the adjustment from unlabelled data: the expectation-maximisation procedure of Saerens and colleagues, or black-box shift estimation. Two things stop that rescuing you here. It needs the source prevalence, which TypeSafe does not publish, and it assumes the positives still look the same, which Michigan's did not. COVID changed what a septic-looking patient was, not merely how many there were. And prior-shift correction never recovers the sign of the model's own miscalibration, which is the part four teams could not agree on.

Ordering vs magnitude: what the split looks like in code

python
from langchain_typesafe import Choice, Noul, Score, TypeSafeClassifierclassifier = TypeSafeClassifier()QUEUES = {    "billing": "Charges, invoices, refunds, payment methods.",    "technical": "Errors, outages, integration and API problems.",    "account": "Login, permissions, seats, plan changes.",    "other": "Anything not clearly in the queues above. Goes to general triage.",}# Fitted on 600 labelled POSITIVES from THIS queue, at THIS queue's base rate.# Refit when the alias moves, or when the shadow-mode firing rate# diverges from finance's issued-refund rate (see the detection section).REFUND_MAP = load_calibration_map("refund_gate_v3.json")# The map gives a probability. The boundary comes from the cost ratio:# a wrong auto-refund costs ~3x a wrong escalation (money out vs 4 min of# agent time), so p* = cost_fp / (cost_fp + cost_fn) = 0.75.REFUND_FLOOR = 0.75                      # a calibrated probability, not a raw scoredef route(ticket_state: str) -> str:    """Order-load-bearing. Argmax, and no threshold is needed."""    answer = classifier.invoke(        state=ticket_state,        questions={            "queue": Choice(                instructions="Which specialist queue fits this ticket best?",                options=QUEUES,            )        },    )    picked = answer.choices["queue"]    log.info("route", queue=picked.choice, dist=picked.distribution,             confidence=picked.confidence)   # ordering class: log, never gate    return picked.choicedef rank_pending_deletions(states: list[str]) -> list[str]:    """Also order-load-bearing, and the blast radius is enormous.    A human works down this list and presses the button. No threshold,    no calibration map, and none needed: nothing compares a number to a    constant. Consequence is high and it does not change the placement.    """    scored = []    for st in states:        answer = classifier.invoke(            state=st,            questions={                "risk": Score(                    instructions="How irreversible is this deletion?",                    levels=["trivial", "recoverable", "costly", "permanent"],                )            },        )        scored.append((answer.scores["risk"].score, st))    return [st for _, st in sorted(scored, reverse=True)]def refund_gate(ticket_state: str) -> str:    """Magnitude-load-bearing. The raw number is not admissible here."""    answer = classifier.invoke(        state=ticket_state,        questions={            "refund_ok": Noul(                instructions=(                    "Does this ticket satisfy every condition in the refund "                    "policy quoted in the state? Answer no if the state does "                    "not contain the policy."                ),            )        },    )    raw = answer.nouls["refund_ok"].noul     # bare float; Nouls carry no confidence    # Monotone, so the weak ordering survives. Isotonic creates ties, so    # never consume a margin downstream of this.    p = REFUND_MAP.transform(raw)    if p < REFUND_FLOOR:        return "human"    receipt = issue_refund(ticket_state)    return verify(receipt)                   # the decision is not the outcome

log, load_calibration_map, issue_refund and verify are yours, and the "human" branch assumes you have somewhere to put the overflow: an approval path that does not collapse under its own queue. Every uncalibrated gate you leave in place is volume aimed at that queue. The third function is the one that carries the argument. rank_pending_deletions can destroy data permanently and route cannot, so on a consequence axis they sit at opposite ends. They are written identically anyway: argmax, no threshold, no map, ships unfitted. refund_gate sits between them in blast radius and is the only one of the three that needs a fit, because it is the only one where a number meets a constant.

Note the fitting budget in that comment, because it is the real price of the magnitude class. Count positives, not rows. Two hundred positives is a working floor for a two-parameter Platt fit and isotonic wants more, so on a queue running at 3% that is several thousand labelled tickets. This is worst exactly where the article said the need is greatest, since the rare-positive queues that calibrate worst are also the ones that take longest to accumulate positives. When a position can be arranged to consume an ordering instead, that cost disappears, which is the practical reason the partition is worth drawing at all.

The last line carries its own point. Retriever AI logged Jev selecting a "Send" action at 0.95 probability and 0.94 confidence, while the browser reported no observable effect. The action had in fact succeeded, which makes the case cleaner rather than weaker: a confident answer to what should I do was read as evidence about did it work. Those are different questions, and only one of them was asked.

Confidence, the winner's probability, and the margin over the runner-up are three different measurements, and nothing forces them to move together. Retriever AI's run happens to show it: two contests, both won at 0.22, margins of 0.08 and 0.01, and the closer one carried the higher confidence. That is two observations and proves nothing on its own, but the reason it can happen is structural rather than incidental. Log all three.

Which agent-harness positions need a fitted threshold

mermaid
flowchart TD
    A[New harness position] --> G{What does the code<br/>read off the answer?}
    G -- Only the winner --> C[Order-load-bearing]
    G -- A difference between<br/>two options --> M[Margin-load-bearing]
    G -- The number itself --> D{Was the constant fitted<br/>on labelled outcomes<br/>from THIS queue?}
    D -- Yes --> E[Magnitude-load-bearing<br/>and fitted]
    D -- No --> F[Threshold Transplant]

    C --> C1[Ship it. Use argmax.<br/>Log the distribution,<br/>the winner and the margin.]
    M --> M1[Fit on the margin, not the score.<br/>A per-score map will not<br/>repair a difference.]
    E --> E1[Ship it. Pin the model version.<br/>Refit when the firing rate<br/>leaves the base rate.]
    F --> F1[Do not ship.<br/>Fit a map, or fall back<br/>to ordering plus a human.]

    style A fill:#4A90E2,color:#FFFFFF
    style G fill:#7B68EE,color:#FFFFFF
    style D fill:#7B68EE,color:#FFFFFF
    style C fill:#6BCF7F,color:#2C2C2A
    style E fill:#6BCF7F,color:#2C2C2A
    style M fill:#FFA07A,color:#2C2C2A
    style F fill:#E74C3C,color:#FFFFFF
    style C1 fill:#98D8C8,color:#2C2C2A
    style M1 fill:#98D8C8,color:#2C2C2A
    style E1 fill:#98D8C8,color:#2C2C2A
    style F1 fill:#FFD93D,color:#2C2C2A

The diagram asks the question in the order a reviewer can actually answer it. "Is this action dangerous" invites a judgement, while "what does the code read off the answer" is settled by looking at the line below the call.

Positions that pass on ordering alone, and need no fit:

  • Model routing: LangChain's ModelRouterMiddleware exposes no threshold at all, which is the right instinct whether or not it was deliberate.
  • Candidate pruning: Retriever AI cut an Amazon action menu from 591 actions to 82, and 513 to 55, by scoring each one and keeping the top of the list. Worth naming what that assumes, since the article has just leaned on the opposite point: it compares scores across separate calls, and every published measurement is within a single question. The pruning worked. It is still resting on something nobody has tested.
  • Triage order and escalation ordering, where a human sees the queue and the ranking decides who they see first.
  • Retrieval reranking, where the output is a sorted list.

The other class is everything that blocks or permits, so a tool-call admissibility check that stops execution belongs there. So does any auto-approval of a write, a payment, a deletion, or an outbound message, and any auto-close path that skips human review. The worst position in the class is an alerting threshold on a rare event: that is where Pervez measured the error growing, and it is where Michigan ended up.

Three rules fall out of the split:

  1. Pin the model version anywhere a threshold exists, because jev-latest moves silently and a fitted map is a fit against specific weights.
  2. Fit per primitive. A threshold tuned on a Noul does not transfer to a Choice, on TypeSafe's own evidence.
  3. Verify the outcome in code, since no probability about an action is evidence the action happened, which is the gap consequence modeling exists to close.

One caveat sits underneath all three, and it is unresolved rather than answered: nobody has tested whether a fitted calibration map transfers across domains. The question was opened on Sacco's repository on 18 September and is still open. If the answer turns out to be no, a map fitted on your refund queue will not serve your deletion gate either, and the cost of the magnitude class goes up per position.

What would make Jev's confidence numbers trustworthy

TypeSafe could ship calibration guarantees, or per-base-rate guidance, and collapse the magnitude class into the ordering class for most teams. They have not, and calibration is absent from their published defect list. Alternatively the open cross-domain question resolves favourably, one map serves many positions, and fitting becomes cheap enough to stop being an architectural concern.

Until one of those happens, the split holds, and it is not really about Jev. Any model that hands back a number and an ordering hands back two products with different warranties. Jev makes the split legible because it returns nothing else: no prose to hide behind, no rationale to over-read, just a typed value and a float. Decisions without generation are decisions with the scaffolding removed. What the scaffolding hid is that the number and the ordering were never the same promise.

References

Agentic AI

Follow for more technical deep dives on AI/ML systems, production engineering, and building real-world applications:

Get the next article by email

One email when a new piece goes up. No digest, no drip sequence.

One email per new article. Unsubscribe in one click.

Books by Ranjan Kumar

The 7 GenAI Architectures cover

The 7 GenAI Architectures

Explore the Book
Building Real-World Agentic AI Systems with LangGraph cover

Building Real-World Agentic AI Systems

Explore the Book
The ChatML Handbook cover

The ChatML Handbook

Explore the Book
The Chat Templates Handbook cover

The Chat Templates Handbook

Explore the Book

Comments