A customer was owed 120.00, and the billing agent proposed refunding 940.00. Account acct_4417 had been charged twice for the SSO add-on on invoice inv_9107. 940.00 appears on no invoice row. What follows is what goes wrong when a structured-output retry sends the validation error back to the model: for a broken JSON shape the re-ask is safe, and for a wrong value it coaches the model.
The harness caught it. A grounding comparator looked for 940.00 in the tool results, found nothing, and the repair loop sent the failure back to the model: amount: 940.00 appears in no tool result; expected: an amount from an invoice row. On the second attempt the model proposed 960.00, the team plan row, a charge nobody disputed. That amount appears in the tool result, so the grounding check passed, and so did the refundable-balance check from Part 1. That run's repair dashboard logged it as a success on the first re-ask. (The case is constructed, from the billing support agent this series is built around. It is not the September run from Part 6, which escalated its 940.00.)
Nothing in that trail was false. The check passed, and 960.00 is in the tool result. What changed between the two attempts is what a pass could tell you. On the first attempt, a model that had misread the invoice could fail the check, and it did. On the second, the correction had listed what the check accepts. Passing then showed that the model could follow an instruction. It no longer showed that the model had found the right charge.
This part's claim is that a validation pass is evidence of correctness only to the degree that the model's likely wrong answers would fail it. The disclosure test applies that to a correction before it is sent, in two parts: the correction must say nothing about which values pass, and every value the model could write in reply must be checked again. A re-ask that carries a content correction tells the model what the comparator accepts, and I call the pass that follows a coached pass. A shape correction, such as a code fence, a trailing comma or truncated JSON, is safe to send back. It says nothing about which values pass, and every value in the new attempt is checked again. So the harness re-asks for shape only. Content failures go to a narrower question whose answer code can check, or to a rung that stops and says it could not produce a result. Every proposal carries the rung it passed at, so the gate can tell the model's own answer from a coached one.
The idea underneath is old. Program-repair research has called it plausible versus correct for a decade: most patches that pass a test suite are not correct fixes (Qi et al., 2015). Pan et al. named the language-model version in-context reward hacking, and output refinement is one of the two feedback loops they show it in (Pan et al., 2024). ImpossibleBench measured it on coding agents in 2025 (Zhong et al., 2025). Libraries already ship pieces of the remedy. Guardrails lets you keep a validator out of the re-ask, and PydanticAI records why each retry happened. None of them makes the split by disclosure the default, and none hands the gate the rung a proposal passed at as a grade of its evidence. That second piece is this part's main contribution. The gate should know whether a number is the model's own, the repair code's, computed from evidence, or the result of a correction, and the disclosure test is the rule that keeps the last kind out. It sits on the book's Repair Ladder, and that ladder's top rung is the book's Deferral Rung.
Why a structured output retry that passes can still be wrong
In Part 3's Role Test, a comparator emits a verdict against a standard. What that verdict says about correctness depends on how wrong answers fare against it. If most wrong answers fail, a pass is strong evidence. If every answer the model is likely to give passes, a pass tells you only what the check tests.
Before the correction, the grounding check accepted 2 of the amounts a model could write: 960.00 and 120.00. Both rows were in context, so landing on one of them was not luck. It meant the model had read the invoice, and landing on 120.00 meant it had found the duplicated charge. A model that misread the ticket could land anywhere, as this one did with 940.00. After a correction that says "an amount from an invoice row", landing on a row is only doing what the model was told. The grounding pass still proves the number came from a tool result. It stops proving that the model found the number by itself, and in this check that was the main thing tying "grounded" to "right".
grid-rows: 1
grid-gap: 40
classes: {
step: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
fail: {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
bad: {style: {fill: "#E74C3C"; stroke: "#A93226"; font-color: "#FFFFFF"}}
told: {style: {fill: "#FFD93D"; stroke: "#C9A700"; font-color: "#2C2C2A"}}
}
before: "First attempt: uncoached" {
label.near: top-center
style: {fill: "#F4F6F7"; stroke: "#95A5A6"}
m: "model writes 940.00\nfrom its own reading" {class: step}
c: "grounding check\naccepts only 960.00, 120.00" {class: step}
v: "FAIL\na misreading could fail,\nso a pass would mean the model read it" {class: fail}
m -> c -> v
}
after: "After the content correction" {
label.near: top-center
style: {fill: "#F4F6F7"; stroke: "#95A5A6"}
m: "model is told:\nuse an amount from an invoice row" {class: told}
c: "model picks 960.00\nfrom 960.00, 120.00" {class: step}
v: "PASS\nproves grounded, not right:\nthe model was told where to look" {class: bad}
m -> c -> v
}
The left side is a check doing its job: the function that rejected 940.00 would have accepted 120.00, and the gap between those outcomes is the evidence. On the right, the correction has moved the acceptance set into the prompt, so no answer from the disclosed set can end in a FAIL. The check's code is the same in both halves. Only the prompt changed. The colours are deliberate: green marks the check doing its job, so the left FAIL is the good outcome and the right PASS is the bad one.
Research on self-correction says the same thing from the other side. Huang et al. found that "LLMs struggle to self-correct their responses without external feedback, and at times, their performance even degrades after self-correction" (Huang et al., 2024). Kamoi et al.'s survey adds the concession this argument has to make: "self-correction works well in tasks that can use reliable external feedback" (Kamoi et al., 2024). A grounding comparator is reliable external feedback.
But it is reliable about whether a number came from a span, and silent on whether the refund is right. The coached pass lives in that gap. Stechly, Valmeekam and Kambhampati report "significant performance gains with sound external verification" (Stechly et al., 2024), but their verifiers check the correctness criterion itself. When the check is correctness, a coached pass is fine. Billing refunds have no such check, which is why the book's running example keeps failing on a number.
Outside billing, the cleanest measured case I know is in circuit design. Hedayati Pirouzan ran model repair against a topology validator for LLM-generated circuits. Repair "raised topological validity from 40.0% to 84.4% (+20 circuits, no regressions)" while moving component agreement by 2 circuits (Hedayati Pirouzan, 2026). One repair "closed a floating node by adding a resistor": the validator was satisfied, "while the electrical property that connectivity stands in for is gone." The pass was bought by changing the circuit to suit the validator. ImpossibleBench shows the agent version. Letting agents submit repeatedly raised the open-test pass rate "from 80% to 83%", and raised the cheating rate on its conflicting variant "from 33% to 38%" (Zhong et al., 2025). More feedback bought more passes of both kinds.
What Instructor, Guardrails, PydanticAI and LangChain send on a re-ask
Four widely used libraries build in the re-ask, and none of them draws the line by what a correction discloses. Guardrails lets you set on_fail per validator, so you can keep a content validator out of the re-ask by hand. No library makes that the default.
| Library (checked 2026-11-13) | What the re-ask sends | Default | On exhaustion |
|---|---|---|---|
Instructor (1.17 release, main source) | the previous assistant message plus "Please correct the function call; errors encountered:\n{e}" | max_retries=3 on create, no backoff | raises InstructorRetryException |
| Guardrails AI 0.11 | a template with ${previous_response} and the validator messages; separate wording for unparseable JSON | 1 reask, and only for validators set to on_fail="reask"; a validator with no on_fail raises (OnFailAction.EXCEPTION, validator_base.py at v0.11.0) | not checked for this article |
| PydanticAI 2.68 | the ModelRetry message or the ValidationError details | 1 output retry | raises UnexpectedModelBehavior |
| LangChain classic parsers | prompt, completion and error, "Please try again" | max_retries=1 | re-raises OutputParserException |
Instructor's documentation makes the design choice explicit. It says self-critique is better seen as "validation errors with clear error messages that the system can use to self-correct", and it lists "Pydantic Validation Errors (code or llm based)" next to "JSON Decoding Errors" as the two things retries defend against (Instructor, Reask Validation). Both go through one exception branch, one budget and one template. Its own QuoteExtraction example re-asks with "The quote must be an exact substring from the source text." That is a grounding check, and the correction hands the model the acceptance set: any exact substring.
Guardrails AI and PydanticAI come closer than the rest. Guardrails words a parse-failure re-ask differently from a value-failure re-ask (Guardrails source, constants.xml), and its FIX_REASK action tries a static fix before asking. PydanticAI records a cause on every retry, 'validation_error' for the schema and 'model_retry' for your own validator, along with the attempt number (PydanticAI, Retries). That is the right data, but PydanticAI still spends one budget on both kinds and sends both back to the model.
The wrong way: re-ask until the checks pass
Here is the loop in about 40 lines, with no library, so the mechanism is visible. Its types come from Part 2: an Origin on every span, and the same inv_9107 rows with the SSO add-on charged twice.
from __future__ import annotationsimport jsonimport refrom dataclasses import dataclassfrom decimal import Decimal, InvalidOperationfrom enum import Enumclass Origin(str, Enum): # Part 2 OPERATOR = "operator" CUSTOMER = "customer" RETRIEVED = "retrieved" TOOL_RESULT = "tool-result" MODEL = "model"@dataclass(frozen=True)class Span: origin: Origin text: str@dataclass(frozen=True)class Context: spans: tuple[Span, ...] = () def add(self, origin: Origin, text: str) -> Context: return Context(self.spans + (Span(origin, text),))INVOICE_ROWS = ( # Part 2's inv_9107: the SSO add-on was charged twice ("inv_9107", "team plan", Decimal("960.00")), ("inv_9107", "sso add-on", Decimal("120.00")), ("inv_9107", "sso add-on", Decimal("120.00")),)CONTEXT = ( Context() .add(Origin.OPERATOR, "Refund only a duplicated charge. " "Refunds above 500.00 need approval.") .add(Origin.CUSTOMER, "I was charged twice for SSO in October. Please fix it.") .add(Origin.TOOL_RESULT, "\n".join( f"row {i}: {inv} {sku} {amount}" for i, (inv, sku, amount) in enumerate(INVOICE_ROWS, 1))))CHANNEL_ACCOUNT = "acct_4417" # Part 4: identity comes from the channel, not the modelBoth comparators follow the book's change to Part 3's verdict: they return corrections that say what should change, not a boolean. Each correction also carries a fix field, which the next section defines. For the wrong way it is unused.
class Fix(str, Enum): SHAPE = "shape" # says nothing about which values pass; all re-checked CONTENT = "content" # discloses what passes, or adds a value nothing checks@dataclass(frozen=True)class Correction: path: str problem: str expected: str fix: FixREQUIRED = ("tool", "account_id", "amount")CONTENT_CHECKED = set(REQUIRED) # every field a content check re-verifiesALLOWED_TOOLS = {"issue_refund"}MONEY = re.compile(r"^\d+\.\d{2}$", re.ASCII) # no fullwidth digitsMONEY_IN_TEXT = re.compile(r"(?<![\d.])\d+\.\d{2}(?![\d.])", re.ASCII)def check_shape(raw: str, checked: set[str] | None = None ) -> tuple[dict | None, list[Correction]]: """Comparator for shape. Reports what should change; never raises.""" try: obj = json.loads(raw) except json.JSONDecodeError as e: return None, [Correction("$", f"not JSON ({e.msg})", "one JSON object", Fix.SHAPE)] if not isinstance(obj, dict): return None, [Correction("$", "not an object", "one JSON object", Fix.SHAPE)] checked = CONTENT_CHECKED if checked is None else checked out = [Correction(k, "absent", "present", Fix.SHAPE if k in checked else Fix.CONTENT) for k in REQUIRED if k not in obj] for k in ("tool", "account_id"): if k in obj and not isinstance(obj[k], str): out.append(Correction(k, f"{type(obj[k]).__name__}, not a string", "a string", Fix.SHAPE)) amount = obj.get("amount") if "amount" in obj and (isinstance(amount, bool) or not isinstance(amount, (str, int, float))): out.append(Correction("amount", f"{amount!r} is not a number", "a decimal amount", Fix.SHAPE)) # null lands here elif "amount" in obj and not MONEY.match(str(amount)): out.append(amount_correction(amount)) return obj, outdef amount_correction(amount) -> Correction: """Echo the model's own value back only when echoing changes no digit.""" try: value = Decimal(str(amount)) except InvalidOperation: return Correction("amount", f"{amount!r} is not a number", "a decimal amount", Fix.SHAPE) if not value.is_finite() or value <= 0: # fixing this changes the answer return Correction("amount", f"{amount!r} is not a positive amount", "a positive decimal amount", Fix.CONTENT) try: exact = value == value.quantize(Decimal("0.01")) except InvalidOperation: # too many digits to be an amount return Correction("amount", f"{amount!r} is too large", "a decimal amount", Fix.CONTENT) if not exact: # rounding would pick a value return Correction("amount", f"{amount!r} has more than two decimal places", "a decimal string with exactly two places", Fix.SHAPE) return Correction("amount", f"{amount!r} is not a decimal string", f"the same amount as the string '{value:.2f}'", Fix.SHAPE)def grounded_amounts(ctx: Context) -> set[Decimal]: """Amounts the system was shown in a tool result. Not operator or customer text.""" found: set[Decimal] = set() for span in ctx.spans: if span.origin is Origin.TOOL_RESULT: found |= {Decimal(m) for m in MONEY_IN_TEXT.findall(span.text)} return founddef check_content(obj: dict, ctx: Context) -> list[Correction]: """Comparator for content. Every field the action uses is checked here.""" out = [] if obj["tool"] not in ALLOWED_TOOLS: out.append(Correction("tool", f"{obj['tool']!r} is not allowed", "an allowed tool", Fix.CONTENT)) if obj["account_id"] != CHANNEL_ACCOUNT: out.append(Correction("account_id", "not the ticket's account", "the ticket's account", Fix.CONTENT)) amount = Decimal(obj["amount"]) if amount not in grounded_amounts(ctx): out.append(Correction("amount", f"{amount} appears in no tool result", "an amount from an invoice row", Fix.CONTENT)) return outdef evaluate(raw: str) -> tuple[dict | None, list[Correction]]: obj, corrections = check_shape(raw) if obj is None or corrections: return obj, corrections obj = {k: obj[k] for k in REQUIRED} # fields no check reads never reach the gate return obj, check_content(obj, CONTEXT)grounded_amounts reads only Origin.TOOL_RESULT. An operator's "above 500.00" is not evidence about this account, which is Part 2's point about reach. Neither is a number the customer typed, so a customer who writes "you owe me 999.99" does not ground 999.99.
My stand-in model below does what a correction says, literally, and picks the largest grounded amount. Any pick from the set would leave the same trail. It shows the mechanism, not a rate, and this article measures no real model's behaviour after coaching. The stand-ins see only the correction text. A real re-ask in this design carries the original context and the corrections, but not the previous attempt, which reverses the April post's advice to include the bad output. The argument needs only that a coached model may pick any row and still pass, so a pass can no longer tell you which kind of model you had.
FIRST = '{"tool": "issue_refund", "account_id": "acct_4417", "amount": "940.00"}'def refund(amount: str) -> str: return json.dumps({"tool": "issue_refund", "account_id": "acct_4417", "amount": amount})def literal_model(prompt: str) -> str: """Stand-in for a model that does what a correction says, to the letter.""" if "expected: an amount from an invoice row" in prompt: return refund(str(max(grounded_amounts(CONTEXT)))) # any row passes if "list the invoice rows" in prompt: return '{"rows": [1]}' return FIRSTdef render(corrections: list[Correction]) -> str: lines = [f" {c.path}: {c.problem}; expected: {c.expected}" for c in corrections] return "The previous response could not be used.\n" + "\n".join(lines)def reask_until_valid(raw: str, model, max_retries: int = 3): """The usual loop: re-ask with every correction until the checks pass.""" sent: list[Correction] = [] for attempt in range(max_retries + 1): obj, corrections = evaluate(raw) if not corrections: return obj, attempt, sent if attempt == max_retries: raise RuntimeError(f"max retries exceeded: {corrections[0].problem}") sent += corrections raw = model(render(corrections))def reader_model(prompt: str) -> str: """Stand-in for a model that read the ticket and, told, picks the duplicate.""" if "expected: an amount from an invoice row" in prompt: return refund("120.00") return FIRSTobj, reasks, sent = reask_until_valid(FIRST, literal_model)right, right_reasks, _ = reask_until_valid(FIRST, reader_model)billed = sum(amount for _, _, amount in INVOICE_ROWS)for name, o, n in (("literal_model", obj, reasks), ("reader_model", right, right_reasks)): a = Decimal(o["amount"]) print(f"{name}: passed after {n} re-ask(s), refund {a}, " f"grounded {a in grounded_amounts(CONTEXT)}, within balance {a <= billed}")print("content corrections sent to the model:", [c.path for c in sent if c.fix is Fix.CONTENT])literal_model: passed after 1 re-ask(s), refund 960.00, grounded True, within balance Truereader_model: passed after 1 re-ask(s), refund 120.00, grounded True, within balance Truecontent corrections sent to the model: ['amount']One model misread the ticket and the other read it. After the content correction their trails are identical: one re-ask, every check passed. Only the amounts differ, and nothing in the loop's record says which amount is owed. Here is where it went wrong.
direction: down
classes: {
step: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
check: {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
warn: {style: {fill: "#FFD93D"; stroke: "#C9A700"; font-color: "#2C2C2A"}}
bad: {style: {fill: "#E74C3C"; stroke: "#A93226"; font-color: "#FFFFFF"}}
}
model: "model\nproposes 940.00" {class: step}
checks: "shape + grounding checks" {class: check}
render: "render every correction\nshape and content alike" {class: warn}
pass: "pass on attempt 2: 960.00\nlogged as a successful repair" {class: bad}
model -> checks: "1. attempt"
checks -> render: "2. amount fails"
render -> model: "3. expected: an amount\nfrom an invoice row"
checks -> pass: "4. every check passes"
Step 3 is the defect. The arrow back to the model carries the comparator's acceptance criterion, and step 4 then reports the comparator's agreement as if it were independent. With Part 1's checks alone, the 960.00 refund goes out. Even the operator's "refunds above 500.00 need approval" does not hold it, because that rule is prompt text, a sensor in Part 3's terms, not a gate. A coached value under 500.00 would pass every check here too. Part 2's duplicate cap would stop it, but its message, "over the duplicated amount", blames the number. The fact that matters is that the model did not know which charge was duplicated, and the gate can no longer read it.
The disclosure test: which corrections can go back to the model
"Shape" and "content" feel obvious until a missing field or a truncated response arrives, so the line between safe and unsafe re-asks needs a rule that code can apply.
A correction may go back to the model only if it says nothing about which values would pass a content check, and every value the model could add or change in reply is checked again on the new attempt.
That is the disclosure test. Its first half stops coaching: not JSON and absent say nothing about which amounts pass, while expected: an amount from an invoice row says exactly that. Its second half stops invention.
A truncated response passes, because completing it means writing new values, and check_content checks every field on the new attempt: the tool against an allowlist, the account against the one the channel assigned, the amount against the tool results. A missing field passes only if one of those checks reads it. A field nothing checks fails, because a re-ask would make the model invent a value no one verifies. A correction that passes both halves is Fix.SHAPE. One that fails either is Fix.CONTENT, and the code above files it that way through CONTENT_CHECKED. So a shape re-ask is only as safe as your content checks are complete: any field the action uses and no check reads can change silently on a re-ask.
Behind the first half is a number, the share of the model's likely wrong answers that the checks would refuse. Call it the estimated rejection rate. It is estimated because it rests on a list of likely wrong answers rather than on a measured distribution. A correction passes the first half when sending it leaves that rate where it was. Its second half is a coverage rule, not a number. A value no check reads has a rejection rate of zero whatever you send, so the question there is not whether the rate moved but whether a check exists at all. not JSON changes nothing about which amounts a confused model writes. "An amount from an invoice row" moves every likely wrong answer into the set the check accepts.
def rejection_rate(wrong: tuple[str, ...]) -> float: """Share of wrong answers the content checks would refuse.""" refused = [bool(check_content(json.loads(refund(a)), CONTEXT)) for a in wrong] return sum(refused) / len(wrong)LIKELY_WRONG = ("940.00", "1200.00", "960.00", "240.00") # misread, total, row, bothprint(f"uncoached: the checks refuse {rejection_rate(LIKELY_WRONG):.2f} " f"of {LIKELY_WRONG}")disclosed = sorted(str(a) for a in grounded_amounts(CONTEXT)) # "an invoice row"coached = tuple(a for a in disclosed if a != "120.00") # wrong answers left to pickprint(f"coached: the wrong answers left are {coached}, " f"refused {rejection_rate(coached):.2f}")_, unchecked = check_shape('{"tool": "issue_refund", "amount": "120.00"}', checked={"tool", "amount"})print("account_id missing, no account check:", unchecked[0].fix.value)for a in ("940.0", "959.999", "-120.00"): c = amount_correction(a) print(f"amount {a}: {c.fix.value}, expected {c.expected}")uncoached: the checks refuse 0.75 of ('940.00', '1200.00', '960.00', '240.00')coached: the wrong answers left are ('960.00',), refused 0.00account_id missing, no account check: contentamount 940.0: shape, expected the same amount as the string '940.00'amount 959.999: shape, expected a decimal string with exactly two placesamount -120.00: content, expected a positive decimal amountThe four likely wrong answers are a judgement, not a measurement: a hallucinated amount, the invoice total, the wrong row, and both copies of the duplicate. On your own traffic, take them from the wrong proposals your decision log has already recorded. Uncoached, the checks refuse 3 of the 4. Once the correction has named the acceptable set, a model that follows it picks from that set. The coached line is that set minus the right answer, so its 0.00 holds by definition. Without a line of the check changing, the rate fell from 0.75 to 0.00, and that drop is what makes the next pass a coached one. The account_id line is the second half: with no check on the account, check_shape files a missing one as CONTENT, so the ladder will not ask the model to invent it.
Three amount lines follow, and they apply the first half to the float correction. A value with exactly two places is echoed back as itself. 959.999 is not, because rounding it to 960.00 would hand the model a grounded value, so the correction names a format and no number. A negative amount is CONTENT, because no fix keeps its value. evaluate also drops any field outside the schema before the gate reads the proposal, so a re-ask cannot add one that no check reads.
The same number limits the second half. A missing amount is SHAPE here because grounding checks the new value, but grounding refuses only 0.75 of the likely wrong answers, not all of them. A re-asked amount is only as trustworthy as that 0.75, which is why the gate below caps the amount on every path.
What the test measures is what a correction adds beyond what the original prompt already said, and that cuts both ways. If your system prompt says "use only amounts from the tool results", the first attempt is coached in the same sense: the model was told the acceptance set before it answered, and a grounding pass on attempt 1 is already weak evidence of correctness. Telling the model to ground its numbers still helps, because it moves the model's answers toward the evidence. It just stops a grounding check from telling you much afterwards. In this example the operator span says "Refund only a duplicated charge", which states the correctness rule, not the grounding check's acceptance set, so attempt 1 here is uncoached for grounding. Where you can, use the stronger design from the start: ask for the evidence ids on the first call, as narrow does, and keep the ladder for when that fails.
A bare rejection, "that amount was rejected, re-read the ticket", does not escape the test. It discloses one bit per attempt, and with two re-asks and only two grounded amounts on this invoice, a model can find the acceptance set by trial. That is ImpossibleBench's multiple-submission result again. Once the budget can enumerate the acceptable values, a bare rejection is CONTENT too. Sorted by the test, the common corrections fall into the grid below.
grid-rows: 5
grid-columns: 3
grid-gap: 14
classes: {
head: {style: {fill: "#95A5A6"; stroke: "#6E7B7C"; font-color: "#FFFFFF"; bold: true}}
shape: {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
content: {style: {fill: "#E74C3C"; stroke: "#A93226"; font-color: "#FFFFFF"}}
note: {style: {fill: "#F4F6F7"; stroke: "#95A5A6"; font-color: "#2C2C2A"}}
}
h1: "Correction" {class: head}
h2: "Disclosure test" {class: head}
h3: "Where it goes" {class: head}
a1: "code fence, trailing comma" {class: note}
a2: "SHAPE: no value moves" {class: shape}
a3: "deterministic rung, no model" {class: note}
b1: "truncated JSON (not a token-limit stop),\namount sent as 940.0 (float)" {class: note}
b2: "SHAPE: discloses no passing value;\nnew values re-checked" {class: shape}
b3: "re-ask with the correction" {class: note}
c1: "required field absent,\nno check on its value" {class: note}
c2: "CONTENT: filling it adds\na value nothing checks" {class: content}
c3: "fill from the system of record,\nor defer" {class: note}
d1: "amount not grounded,\nbusiness rule broken" {class: content}
d2: "CONTENT: the fix\nis the answer" {class: content}
d3: "narrow by derivation,\nor defer" {class: note}
The second row's float correction echoes the model's own value, '940.00', never a value the model did not write. Writing "like '120.00'" there would have coached the right answer into a wrong model through a correction that looks harmless. The truncation in that row is one that did not stop at the token limit. A response cut off at max_tokens needs a rule before the ladder runs, covered in the retry section below, and the stand-in truncation in the demos is not that kind.
Row three is the case teams get wrong. A missing required field looks like a shape problem, and every library files it with the schema errors. But filling it adds a value the model did not write, and if no check reads that value, nothing will catch an invented one. Gabriel Anhaia describes the result: when the schema demands fields the broken output lacks, "the repair pass will invent values to satisfy validation" (Anhaia, 2026). Martin Schenk saw it with no re-ask loop involved: a required statistic field produced fabricated statistics, and 27 of 35 had to be removed from live articles (Schenk). For an account id, the fix is not to ask at all. Part 4 assigned identity from the channel, so the harness already knows the account. Fill the field from that record, or better, stop asking the model for it.
Constrained decoding does not remove the test. Anthropic's structured outputs guarantee "No JSON parsing errors" and "Guaranteed field types and required fields" (Anthropic, Structured outputs). That moves most SHAPE rows out of the ladder before they happen, which is the cheapest fix of all. The same page notes that it does not "guarantee the capitalization of string enum and const values", and that the response "completes normally, with no error". A schema that forces a field to exist also forces the model to fill it, which is the third row again. Every CONTENT row is untouched: a valid 960.00 is still valid.
The Repair Ladder: deterministic repairs before any re-ask
The book's Repair Ladder is an ordered set of responses to a failed verdict. Each rung uses more information and costs more than the one before it, and the ladder is declared in a file, so its worst case is readable without running anything. It has five rungs: accept, deterministic, reask, narrow, defer. Ladders of this shape exist elsewhere. Anhaia's four-rung "recovery ladder" ends in a human queue. Guardrails AI's FIX_REASK fuses the deterministic and reask rungs into one action, and its REFRAIN plays the part of a deferral. Order matters more than the name, and the disclosure test decides which failures may climb past the second rung.
direction: down
classes: {
free: {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
call: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
narrow: {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
stop: {style: {fill: "#9B59B6"; stroke: "#6F3A87"; font-color: "#FFFFFF"}}
}
accept: "1. accept\nchecks pass as written: 0 calls" {class: free}
det: "2. deterministic\nfence, trailing comma: 0 calls" {class: free}
reask: "3. reask, at most 2\nSHAPE corrections only" {class: call}
narrow: "4. narrow, at most 1\nask for evidence, compute the value" {class: narrow}
defer: "5. defer\nstop and say so" {class: stop}
accept -> det: "fails"
det -> reask: "still fails"
reask -> narrow: "still fails, or a CONTENT\ncorrection that is never sent"
narrow -> defer: "no usable evidence"
The edge from rung 3 to rung 4 is where this part departs from the book. In the chapter's prose, any failure climbs through reask. The companion code happens to wire only the shape comparator into it, without saying why. Here a content correction skips it. Nothing changes in the worst case: 3 model calls for repair on one validation, 2 re-asks and 1 narrowing, or 4 calls on the turn counting the original.
Rung 2 comes first because it is free and cannot coach anything. It also fixes less than people hope. One 2026 preprint, which I could read only through a mirror and which has not been peer reviewed, ran 3,997 generation attempts on nine OpenAI and Anthropic configurations. Markdown-fenced JSON was the most common failure, and "among 681 fenced failures, only 48.0% pass validation after fence removal" (Verma et al., 2026). Often the fence was hiding a second violation. Stripping it is still right, because it exposes the real failure without spending a call.
A deterministic repair must not change a value
A deterministic repair is the one place the harness rewrites the model's output itself. It passes the disclosure test only if it touches syntax and nothing inside a value. The obvious trailing-comma regex fails that, because it also matches inside strings.
TICKS = "`" * 3 # written this way so the fence cannot close this listingFENCE = re.compile(rf"^\s*{TICKS}[A-Za-z]*[ \t]*\n(.*?)\n\s*{TICKS}\s*$", re.DOTALL)def strip_fence(text: str) -> str: match = FENCE.match(text) return match.group(1) if match else textdef drop_trailing_commas(text: str) -> str: """Drop a comma that comes right before } or ], but never inside a string.""" out: list[str] = [] in_string = escaped = False for i, ch in enumerate(text): if in_string: if escaped: # this character was escaped: keep it escaped = False elif ch == "\\": escaped = True elif ch == '"': in_string = False elif ch == '"': in_string = True elif ch == "," and text[i + 1:].lstrip()[:1] in ("}", "]"): continue out.append(ch) return "".join(out)DETERMINISTIC = ( ("stripped a code fence", strip_fence), ("dropped a trailing comma", drop_trailing_commas),)print(drop_trailing_commas('{"note": "a, ]", "amount": "120.00",}'))print(drop_trailing_commas(r'{"note": "say \", ]", "amount": "120.00",}')){"note": "a, ]", "amount": "120.00"}{"note": "say \", ]", "amount": "120.00"}The string values "a, ]" and "say \", ]" come through untouched, escaped quote included. Only the comma before the closing brace goes. strip_fence matches only a response that is nothing but a fence. Prose before the fence, such as "Here is the JSON:", falls through to a shape re-ask, because guessing where an object starts is not a repair that keeps every value. My book's companion code uses re.compile(r",\s*([}\]])") over the whole text. On the same input it returns "a]", rewriting a value the model wrote. On a refund note nobody would notice. On an address field, a customer would. Each repair's note also goes into the trail, because the model's output was modified and Part 6's decision log has to be able to say by what.
Repair without a re-ask: ask for the evidence, not the number
When the amount fails grounding, the content correction cannot go back. The ladder still has one option cheaper than stopping: a smaller question that discloses no acceptance set for the amount. Ask which invoice rows the refund repays, then compute the amount in code.
direction: down
classes: {
call: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
code: {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
gate: {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
ok: {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
esc: {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
}
ask: "1. ask: which invoice rows\ndoes this refund repay?" {class: call}
rows: "2. model answers rows [n]" {class: call}
sum: "3. code computes the amount\nfrom the ledger rows" {class: code}
gate: "4. gate checks each row:\nis it a duplicated charge?" {class: gate}
r3: "row 3: duplicate\nrefund 120.00" {class: ok}
r1: "row 1: team plan\nescalate, naming the row" {class: esc}
ask -> rows -> sum -> gate
gate -> r3
gate -> r1
At step 2 the model can still be wrong. What changes is the kind of claim it makes. "Refund 960.00" is a number that may not map back to one piece of evidence: on an invoice with two 120.00 seats, a bare 120.00 could be the duplicate or a real seat. "Row 1" is a claim about evidence, which the gate can check against the property the refund is supposed to have. Code computes the amount at step 3. The model never writes the number. It names evidence, and code turns that into an amount, which is the repair-time version of a signature that never asks the model for a value the system already knows.
class Rung(str, Enum): ACCEPT = "accept" DETERMINISTIC = "deterministic" REASK = "reask" NARROW = "narrow" DEFER = "defer"@dataclass(frozen=True)class Step: rung: Rung max_attempts: int = 1def load_ladder(steps: tuple[Step, ...]) -> tuple[Step, ...]: order = [list(Rung).index(s.rung) for s in steps] if order != sorted(order): raise ValueError("rungs out of order: each one costs more than the last") if not steps or steps[-1].rung is not Rung.DEFER: raise ValueError("the last rung must be defer") return stepsLADDER = load_ladder((Step(Rung.ACCEPT), Step(Rung.DETERMINISTIC), Step(Rung.REASK, 2), Step(Rung.NARROW), Step(Rung.DEFER)))@dataclass(frozen=True)class RepairOutcome: rung: Rung # where the ladder stopped proposal: dict | None # None at defer: an outcome, not an error derived_from: tuple[int, ...] = () # rows the amount was computed from calls: int = 0 # model calls spent on repair trail: tuple[str, ...] = () open: tuple[Correction, ...] = () # what the person picking it up seesdef narrow(obj: dict, model) -> tuple[dict | None, tuple[int, ...]]: """Ask for the evidence, not the number. Code computes the amount.""" reply = model("list the invoice rows this refund repays, as {\"rows\": [n]}, " "or {\"rows\": []} if no row is a duplicated charge") try: rows = tuple(json.loads(reply)["rows"]) except (json.JSONDecodeError, KeyError, TypeError): return None, () if (not rows or len(set(rows)) != len(rows) # a repeated row is not evidence or any(type(r) is not int or not 1 <= r <= len(INVOICE_ROWS) for r in rows)): # type(), so True is not row 1 return None, () amount = sum(INVOICE_ROWS[r - 1][2] for r in rows) derived = {"tool": obj["tool"], "account_id": obj["account_id"], "amount": str(amount)} # known fields only; extras are dropped return derived, rowsnarrow returns nothing rather than a guess when it cannot use the reply. That covers a reply that is not JSON, rows that do not exist, a row named twice, which would sum the same charge twice, and true, which Python would otherwise accept as row 1. It gets one attempt. Narrowing more than once is how a repair ladder turns into a conversation.
The Deferral Rung: an outcome, not an exception
The book's Deferral Rung is the terminal rung, where the system reports that it could not produce a valid result and stops. A ladder without one ends in either an answer or an exception, and neither says "I could not." A system that must produce an answer guesses when it should defer.
Three of the four libraries in the table end their loop with an exception: InstructorRetryException, UnexpectedModelBehavior, OutputParserException. Guardrails AI offers REFRAIN as an on_fail action, which returns an empty value. That is the closest any of them comes to a deferral rung. What Guardrails does when its reasks run out was not checked for this article. The April version of this series raised too, with raise MaxRetriesExceeded after 3 attempts (Validation Layer Design). An exception is the wrong type for this, because it travels to whatever handler is nearest.
A LangGraph node given a RetryPolicy uses default_retry_on, which refuses to retry ValueError, RuntimeError, LookupError and a few others, and retries every other exception (LangGraph source, _retry.py). Instructor's InstructorRetryException derives from Exception through InstructorError, so it is retried: the whole repair loop runs again, 3 attempts in all, at 3 times the budget its code declares. PydanticAI's UnexpectedModelBehavior is a RuntimeError and LangChain's OutputParserException is a ValueError, so neither is. Whether your repair budget gets multiplied depends on a class hierarchy two libraries away. Part 5 met the same rule from the other side, when it chose a base class for FloorBreached.
grid-rows: 1
grid-gap: 40
classes: {
step: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
bad: {style: {fill: "#E74C3C"; stroke: "#A93226"; font-color: "#FFFFFF"}}
stop: {style: {fill: "#9B59B6"; stroke: "#6F3A87"; font-color: "#FFFFFF"}}
ok: {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
}
raise: "Exhaustion as an exception" {
label.near: top-center
style: {fill: "#F4F6F7"; stroke: "#95A5A6"}
l: "repair loop\nruns out of attempts" {class: step}
e: "raises\nInstructorRetryException" {class: bad}
h: "RetryPolicy, default retry_on:\nretried, 3 attempts" {class: bad}
l -> e -> h
h -> l: "loop runs again"
}
defer: "Exhaustion as a deferral" {
label.near: top-center
style: {fill: "#F4F6F7"; stroke: "#95A5A6"}
l: "ladder reaches\nits last rung" {class: step}
d: "returns RepairOutcome\nrung=defer, open corrections" {class: stop}
g: "gate routes it:\nleave for a person" {class: ok}
l -> d -> g
}
On the left, the loop's budget is whatever the surrounding handlers multiply it by. On the right, the last rung returns a value that carries the corrections nobody could resolve, and the gate decides what happens next. In a deferral proposal is None, so a fallback has nothing to send. The last invalid attempt is not a partial answer.
Part of the rung is visible to the model. narrow's prompt offers {"rows": []} when no row is a duplicated charge, and an empty list defers. ImpossibleBench measured what that kind of option does. Its authors gave agents an explicit way to stop and flag the task for a human. They found it "quite effective for OpenAI models", with o3's cheating rate falling "from 49% to 12%", while "the effect is much less pronounced for Claude Opus 4.1" (Zhong et al., 2025, section 5.3). On the harness side, the rung changes only routing. Only the visible option can change what the model does.
Here is the runner. The reask rung checks the disclosure test before every attempt, because a shape re-ask regenerates the whole object and the new attempt can fail on content.
def run_ladder(raw: str, model, ladder: tuple[Step, ...] = LADDER) -> RepairOutcome: obj, corrections = evaluate(raw) trail: list[str] = [] calls = 0 for step in ladder: if step.rung is Rung.ACCEPT: if not corrections: return RepairOutcome(Rung.ACCEPT, obj, trail=("accept: passed",)) trail.append(f"accept: {corrections[0].path} {corrections[0].problem}") elif step.rung is Rung.DETERMINISTIC: text = raw for note, repair in DETERMINISTIC: if repair(text) != text: text = repair(text) trail.append(f"deterministic: {note}") if text != raw: raw = text obj, corrections = evaluate(raw) if not corrections: return RepairOutcome(Rung.DETERMINISTIC, obj, calls=calls, trail=tuple(trail)) elif step.rung is Rung.REASK: for _ in range(step.max_attempts): if any(c.fix is Fix.CONTENT for c in corrections): trail.append("reask: skipped, a CONTENT correction stays here") break raw = model(render(corrections)) calls += 1 obj, corrections = evaluate(raw) trail.append(f"reask: {len(corrections)} correction(s) left") if not corrections: return RepairOutcome(Rung.REASK, obj, calls=calls, trail=tuple(trail)) elif step.rung is Rung.NARROW: if obj is not None and all(c.path == "amount" and c.fix is Fix.CONTENT for c in corrections): derived, rows = narrow(obj, model) calls += 1 if derived is None: trail.append("narrow: no usable rows") else: trail.append(f"narrow: amount computed from rows {list(rows)}") late = [c for c in check_content(derived, CONTEXT) if c.path != "amount"] # the gate checks the rows if not late: return RepairOutcome(Rung.NARROW, derived, rows, calls, tuple(trail)) corrections = late trail.append("narrow: derived proposal fails a content check") else: trail.append("narrow: skipped, nothing to derive") elif step.rung is Rung.DEFER: trail.append("defer: stopped rather than guessed") return RepairOutcome(Rung.DEFER, None, calls=calls, trail=tuple(trail), open=tuple(corrections)) raise AssertionError("load_ladder guarantees a defer rung")How the gate reads the rung a proposal passed at
Under Part 3's Role Test the ladder is three things, and it is worth saying which. Its verdicts and trail are comparator output: they report. Its defer is a refusal to produce a proposal, which, like Part 5's floor check, works as a bound on the model call rather than a gate on the refund. Its deterministic and narrowing rewrites change the proposal, so they are Harness Residual and get reviewed as application code. Refusal belongs to the gate, which now has more to read than "passed": it knows the rung, and for a narrowed proposal it knows the evidence the amount was computed from.
def duplicate_groups(rows) -> list[list[int]]: """Part 2's duplicate rule, by row number: identical rows form a group.""" groups: dict[tuple, list[int]] = {} for n, row in enumerate(rows, 1): groups.setdefault(row, []).append(n) return [g for g in groups.values() if len(g) > 1]def gate(outcome: RepairOutcome) -> str: if outcome.rung is Rung.DEFER: return "deferred: left for a person, open corrections attached" amount = Decimal(outcome.proposal["amount"]) groups = duplicate_groups(INVOICE_ROWS) if outcome.derived_from: chosen = set(outcome.derived_from) not_dup = sorted(n for n in chosen if not any(n in g for g in groups)) if not_dup: return f"escalated: model chose row(s) {not_dup}, not a duplicated charge" if any(len(chosen & set(g)) > len(g) - 1 for g in groups): return "escalated: refunds every copy of one charge" if amount > sum((len(g) - 1) * INVOICE_ROWS[g[0] - 1][2] for g in groups): return "escalated: over the duplicated amount" # on every path, derived too return f"refund {amount}"TRUNCATED = '{"tool": "issue_refund", "account_id": "acct_4417", "amount": "120.0'FENCED = TICKS + "json\n" + refund("120.00") + "\n" + TICKSNO_ACCOUNT = '{"tool": "issue_refund", "amount": "120.00"}'FOREIGN = '{"tool": "delete_account", "account_id": "acct_9999", "amount": "-5.00"}'def reads_corrections(prompt: str) -> str: return refund("120.00")def names_rows(*rows: int): def model(prompt: str) -> str: if "list the invoice rows" in prompt: return json.dumps({"rows": list(rows)}) return FIRST return modeldef never_fixes(prompt: str) -> str: return TRUNCATEDcases = ( ("fenced, right amount", FENCED, literal_model), ("truncated, model fixes the shape", TRUNCATED, reads_corrections), ("940.00, model names row 1", FIRST, literal_model), ("940.00, model names row 3", FIRST, names_rows(3)), ("940.00, model names row 2", FIRST, names_rows(2)), ("940.00, model names rows 2 and 3", FIRST, names_rows(2, 3)), ("940.00, model names row 3 twice", FIRST, names_rows(3, 3)), ("truncated, model never fixes it", TRUNCATED, never_fixes), ("account_id missing", NO_ACCOUNT, reads_corrections), ("-5.00, wrong tool and account, row 3", FOREIGN, names_rows(3)),)for name, raw, model in cases: outcome = run_ladder(raw, model) print(f"{name}\n rung {outcome.rung.value}, {outcome.calls} call(s): " f"{gate(outcome)}") for line in outcome.trail: print(f" {line}")print("wrong way, same ticket:", gate(RepairOutcome(Rung.REASK, obj, calls=reasks)))fenced, right amount rung deterministic, 0 call(s): refund 120.00 accept: $ not JSON (Expecting value) deterministic: stripped a code fencetruncated, model fixes the shape rung reask, 1 call(s): refund 120.00 accept: $ not JSON (Unterminated string starting at) reask: 0 correction(s) left940.00, model names row 1 rung narrow, 1 call(s): escalated: model chose row(s) [1], not a duplicated charge accept: amount 940.00 appears in no tool result reask: skipped, a CONTENT correction stays here narrow: amount computed from rows [1]940.00, model names row 3 rung narrow, 1 call(s): refund 120.00 accept: amount 940.00 appears in no tool result reask: skipped, a CONTENT correction stays here narrow: amount computed from rows [3]940.00, model names row 2 rung narrow, 1 call(s): refund 120.00 accept: amount 940.00 appears in no tool result reask: skipped, a CONTENT correction stays here narrow: amount computed from rows [2]940.00, model names rows 2 and 3 rung narrow, 1 call(s): escalated: refunds every copy of one charge accept: amount 940.00 appears in no tool result reask: skipped, a CONTENT correction stays here narrow: amount computed from rows [2, 3]940.00, model names row 3 twice rung defer, 1 call(s): deferred: left for a person, open corrections attached accept: amount 940.00 appears in no tool result reask: skipped, a CONTENT correction stays here narrow: no usable rows defer: stopped rather than guessedtruncated, model never fixes it rung defer, 2 call(s): deferred: left for a person, open corrections attached accept: $ not JSON (Unterminated string starting at) reask: 1 correction(s) left reask: 1 correction(s) left narrow: skipped, nothing to derive defer: stopped rather than guessedaccount_id missing rung reask, 1 call(s): refund 120.00 accept: account_id absent reask: 0 correction(s) left-5.00, wrong tool and account, row 3 rung defer, 1 call(s): deferred: left for a person, open corrections attached accept: amount '-5.00' is not a positive amount reask: skipped, a CONTENT correction stays here narrow: amount computed from rows [3] narrow: derived proposal fails a content check defer: stopped rather than guessedwrong way, same ticket: escalated: over the duplicated amountThe third case and the last line are the same ticket and the same model, and both escalate. What differs is what each escalation tells the person who opens it. The wrong way says the amount was too high, about a proposal every comparator approved, which invites that person to trust the number and adjust the cap. The ladder says the model could not ground its own amount and then picked the team plan row, which tells them exactly what to check.
The fourth case is the same ungrounded 940.00 from a model that named the right row, and it refunds 120.00 without a person. Row 2 refunds too, because the gate treats both copies of the SSO charge as one duplicate group. Naming rows 2 and 3 together escalates, because that refunds the real charge along with its copy. Naming row 3 twice defers, because narrow refuses a repeated row.
Missing account_id is SHAPE only because check_content compares the account the model writes with the one the channel assigned, so a re-ask cannot slip in a different account. Drop that check and the same correction becomes CONTENT. In a real harness the case should never reach the ladder, because the model need not be asked for the account at all.
That last case, -5.00 with the wrong tool and account, is why run_ladder checks the tool and the account again after narrowing. A negative amount fails before check_content runs, so without that second look a derived proposal would carry delete_account and acct_9999 straight to the gate.
These stand-ins show which path each kind of failure takes, not how often a real model takes it. That rate is a measurement for Part 2's Plant Model, on your own model and prompts.
grid-rows: 1
grid-columns: 4
grid-gap: 20
classes: {
free: {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
call: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
narrow: {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
stop: {style: {fill: "#9B59B6"; stroke: "#6F3A87"; font-color: "#FFFFFF"}}
}
a: "accept or deterministic\n\nthe model's own values\n\ngate: cap check,\nact without a person" {class: free}
r: "reask (shape only)\n\nnew attempt, never told\nwhat content passes\n\ngate: cap check,\nact without a person" {class: call}
n: "narrow\n\namount computed\nfrom chosen rows\n\ngate: every row a duplicate?\nact, else escalate naming it" {class: narrow}
d: "defer\n\nno proposal\n\ngate: leave for a person\nwith open corrections" {class: stop}
That grid is the policy the gate enforces, one column per rung. No column says "coached", because the ladder never produces a coached pass. If your harness uses a library loop that can, such as Instructor's llm_validator (Instructor, Reask Validation; see also issue #2056 on injection into its prompt) or any content validator inside response_model, keep the field anyway. Mark a pass coached whenever a content correction was sent on the way to it, and send it to a person with the evidence attached. A narrowed proposal carries rows the gate can check. A coached pass carries nothing, so it never acts alone. An LLM judge's reason is a content correction too. Pan et al. found that when a generator refines against an evaluator, "the evaluator's ratings improve while the generation quality remains stagnant or even decreases" (Pan et al., 2024b).
Write the rung into Part 6's Second Stream for every validation, with the derivation rows when there are any. Part 6 asked whether a refund was ever decided. This part asks something narrower: was the number the model's, the repair code's, or the result of a correction? Two details keep this consistent with Part 6. A narrowed proposal is a new proposal derived from the refused one: give it its own id and a derived_from link, and run its owed decisions again. Otherwise Part 6's unbacked_actions would see a refund acted on after a non-allowing verdict. And the rung goes in a named field with a schema_version bump, not in Part 6's closed Outcome enum.
Repair is not retry, and the April advice on backoff was wrong
Retry is for a call that failed: a 503, a dropped connection, a timeout. Nothing came back, so the right response is to wait and send the same request again. The Anthropic Python SDK already does this: "By default, the SDK retries certain errors twice, with a short exponential backoff" (Anthropic, Python SDK), and the OpenAI SDK also retries twice by default (OpenAI Python SDK). Repair is for a response that arrived with the wrong content. Waiting does not change a fenced block, so the April post's advice to add exponential backoff between repair attempts only added latency. I am reversing it here.
PydanticAI's docs state the split best: "'Retry' means eight different things in an agent run, at eight different layers, and they don't share budgets." On the same page, a worked example has an output retry budget, the OpenAI SDK's own retries and a transport retry stack up until one agent call "can put 3 x 1 x 3 x 2 = 18 requests on the network" (PydanticAI, Retries), and that product is what a spend limit has to be checked against. Instructor's docs warn that "there are two retry settings with the same name" (Instructor, Retrying), then show a recipe that puts RateLimitError, APIConnectionError and ValidationError under one exponential backoff. The recipe does turn off the SDK's own transport retries, so nothing is retried twice, but it still waits between attempts to fix content, where waiting buys nothing.
A real issue shows what the conflation costs. In Instructor issue #2210, a response cut off at the token limit was treated as a validation error and re-asked, with the truncated output appended each time. Prompt tokens went 15, then 450, then 1,227, and the reporter estimated "~590K output tokens and ~920K prompt tokens per failure" at the provider's default output limit (Instructor #2210). A truncation passes the disclosure test, so a re-ask is allowed. Re-asking the same request at the same max_tokens fixes it only if the model happens to write less next time. So the harness needs a rule that runs before the ladder: when the response stopped at the token limit, raise the limit or shorten the request, and never append the cut-off text.
What to measure instead of repair success rate
The April post set targets for repair success: below 70% meant the repair prompts needed work, and above 95% meant you could validate more strictly. That metric rewards coaching. Every content correction sent back to the model raises it, so a team chasing it will add disclosure to the prompt and get a higher number for proposals that mean less. ImpossibleBench reported the same trade in reverse: hiding the tests "reduces cheating success rate to near zero ... but also degrades performance on the original benchmark" (Zhong et al., 2025). Expect the disclosure test to lower your repair success rate. Some of the passes it removes would have been honest, because a coached model can still pick 120.00. You cannot tell those apart from the coached ones, so you lose both kinds, and that is the price of a pass that means something. Chart the rung distribution instead, because a rise on each rung means something different.
grid-rows: 2
grid-columns: 2
grid-gap: 20
classes: {
free: {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
call: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
narrow: {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
stop: {style: {fill: "#9B59B6"; stroke: "#6F3A87"; font-color: "#FFFFFF"}}
}
d: "deterministic repairs rising\n\noutput format drifting: early warning,\nfree to fix" {class: free}
r: "shape re-asks rising\n\na repair the deterministic rung\ndoes not cover yet" {class: call}
n: "narrowings rising\n\nthe model stopped grounding the amount:\nread the prompt and the model id" {class: narrow}
f: "deferrals at zero for weeks\n\nnothing reaches the top,\nor the deferral path is broken" {class: stop}
Anhaia's ladder already puts "rung-distribution" on the dashboard, and so does the book. This part adds two counts. Count narrowings separately from re-asks. A rising narrowing rate is a content problem: the model has stopped finding its amounts in what it read, and no prompt about JSON will fix that. Then count coached passes wherever a library loop can still produce them. Their target is zero.
When is it safe to re-ask an LLM with the validation error?
For a shape correction, yes, within the ladder's budget, provided every field the action uses is checked again on the new attempt and token-limit truncations are handled before the ladder runs. For a content correction, only when the comparator checks correctness itself, as a test suite or a sound verifier does. That is Stechly et al.'s condition, and billing refunds do not meet it.
Three findings would weaken this argument. The first would be a measurement showing that real models, told a content acceptance criterion, choose the correct value from the disclosed set at the same rate as uncoached models choose it unprompted. A coached pass would then be nearly as informative as an uncoached one. The second is a grounding check that verifies the arithmetic behind a value, which would close most of the gap without narrowing. The book's chapter points that way: the model returns the invoice ids it added up, and the comparator redoes the sum. The third is a provider feature that returns per-value citations together with constrained JSON, which would let a comparator check provenance directly. Anthropic's structured outputs return a 400 error if citations are enabled, so for now the two cannot share a call.
The 940.00 refund, run through the ladder
Back to the opening ticket. The re-ask loop sent the model the rule its answer had broken, got 960.00 back, and logged a successful repair. The ladder never sends that rule. It asks which rows the refund repays, computes the amount, and either refunds 120.00 or hands a person the row the model got wrong. Both harnesses ran the same grounding check. When the ladder passes, the pass still means the model found the duplicated charge, and when it cannot, it escalates with the row the model chose, because a validation pass is evidence of correctness only while the model's likely wrong answers can still fail it.
Checklist: re-asks, repairs and deferrals
- Tag every correction
SHAPEorCONTENTwith the disclosure test: does it say nothing about which values pass, and is every value the model could add or change checked again? - Send
SHAPEcorrections back. Never sendCONTENTones, because the pass that follows is a coached pass, unless the comparator checks correctness itself, as a test suite or a sound verifier does. A missing field that no check reads isCONTENT, so fill it from the system of record or defer. - Run deterministic repairs before any model call, and test that they never change a string value.
- Re-run every content check on a re-asked attempt. A shape re-ask regenerates the whole answer, so every field the action uses needs a check, or must be overwritten from the system of record.
- For an ungrounded number, ask for the evidence and compute the value in code, in one attempt. Reject repeated or non-integer evidence ids, and cap the computed amount like any other.
- End the ladder in a deferral that returns a value, never an exception a retry handler can catch. Load the ladder from a declaration that refuses a missing defer rung or rungs out of order.
- Stamp the rung, and any derivation, on what the gate reads. Act on a narrowed proposal only when every row it names passes the gate's own check; otherwise escalate with the rows attached.
- In library loops (Instructor, Guardrails, PydanticAI), keep content validators out of the re-ask path, or mark passes coached and route them to a person.
- Keep transport retries and repair attempts on separate budgets. No backoff between repair attempts.
- Chart rungs per day. Alert on narrowings and deferrals, and on deferrals stuck at zero.
Validation and repair in the Harness Engineering book (Chapter 7)
This is Part 7 of a series that follows Harness Engineering for Production AI Systems one chapter at a time. Chapter 7, "Validation and Repair," declares the Repair Ladder as a TOML policy file with a loader that refuses a missing defer rung. It walks through the three narrowing shapes, from dropping optional fields to asking a closed question. It also uses the decision log to tell a stale schema from a struggling model, and explains why the ladder's loop belongs inside a graph node rather than on a graph edge. Code for the chapter is the ch07-validation-repair tag of the companion repository. Its tests need no model and no API key.
Writing this part found several things for the book's next edition. The chapter's own example re-ask sends the grounding correction "an amount from an invoice row" back to the model, which is the coached pass this part warns about. In the companion code, grounding counts customer text as evidence. Its trailing-comma repair rewrites string values, its whitespace repair is never needed, its narrow rung re-sends the same request as reask, and its amount pattern lets 1940 pass with no amount checked and reads dates in invoice text as amounts.
References
- Anhaia, G. (2026). JSON Mode Still Fails. Here's the Recovery Ladder. DEV Community. https://dev.to/gabrielanhaia/json-mode-still-fails-heres-the-recovery-ladder-55n
- Anthropic. Python SDK. Claude Platform Docs. https://platform.claude.com/docs/en/api/sdks/python
- Anthropic. Structured outputs. Claude Platform Docs. https://platform.claude.com/docs/en/build-with-claude/structured-outputs
- Guardrails AI. guardrails source,
guardrails/constants.xml, v0.11.0. https://github.com/guardrails-ai/guardrails/blob/v0.11.0/guardrails/constants.xml - Guardrails AI. guardrails source,
guardrails/guard.py, v0.11.0. https://github.com/guardrails-ai/guardrails/blob/v0.11.0/guardrails/guard.py - Guardrails AI. guardrails source,
guardrails/validator_base.py, v0.11.0. https://github.com/guardrails-ai/guardrails/blob/v0.11.0/guardrails/validator_base.py - Hedayati Pirouzan, A. (2026). Validation and Simulation Catch Different Errors: Four Levels of Evaluation for LLM-Generated Circuits. arXiv:2609.26830. https://arxiv.org/abs/2609.26830
- Huang, J., et al. (2024). Large Language Models Cannot Self-Correct Reasoning Yet. ICLR 2024. https://arxiv.org/abs/2310.01798
- Instructor. Retry Logic with Tenacity. https://python.useinstructor.com/concepts/retrying/
- Instructor. v1.17.0 release, 2026-09-09. https://github.com/567-labs/instructor/releases/tag/v1.17.0
- Instructor. Validation and Reasking. https://python.useinstructor.com/concepts/reask_validation/
- Instructor. Issue #2056, Security: Retry amplification and LLM validator injection findings. https://github.com/567-labs/instructor/issues/2056
- Instructor. Issue #2210, Retry Causes Concatenation. https://github.com/567-labs/instructor/issues/2210
- Kamoi, R., et al. (2024). When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs. TACL 12. https://arxiv.org/abs/2406.01297
- Kumar, R. (2026). Harness Engineering for Production AI Systems. Chapter 7, "Validation and Repair." https://harness.ranjankumar.in/
- Kumar, R. (2026). harness-engineering-code, tag
ch07-validation-repair. https://github.com/ranjankumar-gh/harness-engineering-code/tree/ch07-validation-repair - LangChain. langchain-classic
RetryWithErrorOutputParser. https://reference.langchain.com/python/langchain-classic/output_parsers/retry/RetryWithErrorOutputParser - LangGraph.
default_retry_on,libs/langgraph/langgraph/_internal/_retry.py. https://github.com/langchain-ai/langgraph/blob/main/libs/langgraph/langgraph/_internal/_retry.py - OpenAI. openai-python README. https://github.com/openai/openai-python
- Pan, A., Jones, E., Jagadeesan, M., Steinhardt, J. (2024). Feedback Loops With Language Models Drive In-Context Reward Hacking. ICML 2024. https://arxiv.org/abs/2402.06627
- Pan, J., et al. (2024b). Spontaneous Reward Hacking in Iterative Self-Refinement. arXiv:2407.04549. https://arxiv.org/abs/2407.04549
- Pydantic. Pydantic AI: Retries. https://pydantic.dev/docs/ai/core-concepts/retries/
- Qi, Z., Long, F., Achour, S., Rinard, M. (2015). An Analysis of Patch Plausibility and Correctness for Generate-and-Validate Patch Generation Systems. ISSTA 2015. https://www.cs.toronto.edu/~fanl/papers/kali-issta2015.pdf
- Schenk, M. (n.d., undated page). A required field made my AI fabricate statistics. DEV Community. https://dev.to/martinschenk/a-required-field-made-my-ai-fabricate-statistics-5d9m
- Stechly, K., Valmeekam, K., Kambhampati, S. (2024). On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks. arXiv:2402.08115. https://arxiv.org/abs/2402.08115
- Verma, A., et al. (2026). Why Does Structured Output Fail? An Empirical Study of Prompt-Enforced Pydantic Schemas Across Nine LLMs. Preprint, read via the deeplearn.org mirror (no arXiv id found). https://deeplearn.org/arxiv/856077/why-does-structured-output-fail
- Zhong, Z., et al. (2025). ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases. arXiv:2510.20270. https://arxiv.org/abs/2510.20270
Related Articles
- AI Agent Guardrail Monitoring: A Refusal Is Not Proof of Life
- LangGraph Guardrails: A Node That Skips the Executor
- AI Agent Decision Logs: Catch the Refund No Control Decided
- LLM Fallback Models Inherit Authority They Never Earned






