AI agent failure attribution decides whether the model, the harness or the seam between them owns an incident, and missing records push that decision toward the model. I gave my book's attribution protocol a run that kept none of the records its four free questions read. No control stream, no eviction record, no classification of the output, no replay. Then I answered the one question that costs a model call. In production that answer comes from a model; here it is supplied by hand, as the chapter's own examples supply it. The protocol named a culprit:
no-records-one-call: model decided by corrected-output-succeeds unknown controls-ran no list of controls the deployed harness should run unknown required-span-absent no eviction record was kept for this run unknown claimed-and-allowed the output was never classified against the repertoire unknown harness-diverges nothing recorded to replay against yes corrected-output-succeeds a correct output through the same harness reached a correct outcomeFour questions about the harness could not be answered. A fifth asked whether a correct model output would have reached a correct outcome, it said yes, and the model was convicted. That output is the book's package at the tag for Chapter 16, run unchanged from a short script of mine, kept_nothing.py, with two long lines wrapped. The chapter's own no-records incident leaves the paid question unanswered and comes back unattributed, correctly. Answering that one question is all it takes to flip it to model, and that is what a team does when it runs a counterfactual or a judge over traces but keeps no control stream and no list of what should have run. Its own policy file explains why the second question runs before anything about the model: "the model cannot be wrong about a constraint it was never shown." That question came back unknown, and the walk convicted the model anyway.
Nothing in the book's test suite catches this. One of its tests, test_replay_agreement_does_not_clear_the_harness, removes the eviction record and asserts that the answer is the model.
Here is the claim this part owns. A record can convict a layer. Only a whole set of questions, each answered against a record kept at the time, can clear one. My book calls that rule the Exoneration Asymmetry, and its protocol applies it to clearing but not to convicting.
That gap matters because a single-layer verdict is how ownership gets assigned. A verdict of model sends the incident to prompt and evaluation work, and it closes the incident for the team that owns the harness. So a verdict must not hand the failure to one owner while an earlier question could still hand it to another. A harness finding stands on harness records whatever the counterfactual later says. A model finding does not. It is uninterpretable while the eviction record is missing, because the model cannot be wrong about a constraint it was never shown, and it is premature while any harness question is open, because a model verdict routes the incident away from the harness team. So the wait leans one way, on purpose. A harness conviction leaves the model question unasked, and that is acceptable: the harness defect needs fixing either way, and question 5 can still be paid for afterwards.
Two rules follow. First, every record a question reads must say whether it was kept, and replay, the decision trail and the control stream need that bit as much as the eviction record does. Second, a conviction waits while any earlier question that could convict a different layer is still unknown. The second rule alone turns the run above into unattributed, which is the honest answer. The first is what changes the book's own incidents, further down. Three-valued answers are the book's; the cross-layer wait and the three kept bits are what this part adds, and with them in place the rate of unattributed answers measures your records, not your system.
AI agent failure attribution: why "which model?" is the wrong first question
AI agent failure attribution is the work of deciding, after an incident, whether the model, the harness around it, or the seam between them owns a failure. This part is about what happens to that decision when records are missing, which is most of the time.
Most postmortems of an agent incident run by elimination. Rule out the obvious harness causes, find nothing, and the model is what remains. Tian Pan describes the result well: "The post-mortem becomes a blame auction. Everyone bids 'the model changed' because it is an unfalsifiable claim that costs nothing to make" (Pan, April 2026). His fix is to refuse "the model" as an answer "until you've ruled out everything else". That is still elimination, and the post does not separate a cause ruled out by a record from a cause ruled out because nobody looked.
Blame does not always flow to the model, either. A second post by the same author describes the opposite flattening, where "The model's inference was wrong" becomes "the runbook was ambiguous" or "the guardrail was missing", because that is what the postmortem template has a row for (Pan, June 2026). Elimination does not pick the true cause. It picks the cause cheapest to name in that room. The fixed walk later in this part leans one way, on purpose. While a harness record is missing it will not convict the model, and it will not convict the harness on a missing decision trail or a replay that never ran. Its one deliberate exception, a missing control signal, is stated below.
I made the same argument from the loop side last summer: "When an agent misbehaves, the first question is not "which model." It is "which layer."" (The Oracle Problem). That post gave a test, "if I fixed the loop ... would this failure survive?", and left it as a thought experiment. Two engineers can imagine the fixed loop and disagree.
A model reading the traces is the scalable alternative on offer. Who&When, the first benchmark for this, reports that "The best method achieves 53.5% accuracy in identifying failure-responsible agents but only 14.2% in pinpointing failure steps" (Zhang et al., ICML 2025). Its definition of the error that caused a failure is counterfactual: an error is decisive if correcting it turns the trajectory into a success. That is the same shape as the chapter's paid question below, and AgenTracer and FALAT build on it with counterfactual replay and with a check of whether correcting a step is enough to recover the outcome (AgenTracer, FALAT). Later work scores better: FALAT reports 46.0% step-level accuracy on algorithm-generated trajectories and 29.1% on hand-crafted ones.
AgentRx builds "an auditable validation log of evidence-backed violations" before an LLM judges anything (Microsoft Research, 2026), and MP-Bench argues that earlier conclusions are "largely driven by limitations in existing benchmark designs" that assume "a single deterministic root cause for each failure" (In et al., 2026). So I do not claim a judge cannot attribute. My narrower claim is that none of the attribution methods or benchmarks I reviewed (Who&When, AgenTracer, FALAT, AgentRx, MP-Bench) has an output for "the logs cannot support an answer".
grid-rows: 1
grid-gap: 40
elim: "Elimination" {
label.near: top-center
grid-columns: 1
grid-gap: 14
style: {fill: "#FFFFFF"; stroke: "#95A5A6"; font-color: "#2C2C2A"}
q1: "Harness cause?\nnothing found" {style: {fill: "#95A5A6"; stroke: "#6F7F80"; font-color: "#2C2C2A"}}
q2: "Context cause?\nnothing found" {style: {fill: "#95A5A6"; stroke: "#6F7F80"; font-color: "#2C2C2A"}}
q3: "Gate cause?\nnothing found" {style: {fill: "#95A5A6"; stroke: "#6F7F80"; font-color: "#2C2C2A"}}
v: "Verdict: the model\n(what is left)" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
}
conv: "Three-valued attribution" {
label.near: top-center
grid-columns: 1
grid-gap: 14
style: {fill: "#FFFFFF"; stroke: "#95A5A6"; font-color: "#2C2C2A"}
a1: "Harness cause?\nno record: unknown" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
a2: "Context cause?\nrecord says no" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
a3: "Gate cause?\nno record: unknown" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
w: "Verdict: unattributed\nkeep these two records" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
}
Both columns ask the same three questions. On the left, "nothing found" covers both a record that says no and a record that was never written, so every row narrows toward the model. On the right, the yellow rows stay open, and the verdict names the records that would close them.
The Exoneration Asymmetry: convicted by a finding, cleared by a kept record
Chapter 16's protocol is a file of five questions, run in order. Four are free because they read records the harness already keeps. Only the fifth costs a model call. Each question convicts one layer when it answers yes, and none of them declares what a no acquits:
| Order | Question | Reads | A yes convicts | Cost |
|---|---|---|---|---|
| 1 | controls-ran | the control stream, Part 6's Second Stream (Part 6) | harness | free |
| 2 | required-span-absent | the eviction record (Part 5) | harness | free |
| 3 | claimed-and-allowed | catches declarations (Part 15) and the decision trail | harness | free |
| 4 | harness-diverges | replay of today's controls | harness | free |
| 5 | corrected-output-succeeds | a counterfactual run | model | one model call |
Every answer is one of three values. no means the record exists and says the thing did not happen. unknown means the record or judgement the question reads does not exist: a log nobody kept, a classification nobody made, a counterfactual nobody paid for. From that split comes the chapter's definition of its first named concept, quoted word for word: "A layer is convicted by a finding and cleared only by a record that was written at the time. The absence of a finding clears nobody." A layer is cleared only when every question that could convict it has answered no against a kept record. If both layers clear and the outcome is still wrong, the case is an interaction. If neither convicts nor clears, it is unattributed.
Testing had this rule long before agents did. Dijkstra put the testing version in 1972: testing "can be a very effective way to show the presence of bugs, but is hopelessly inadequate for showing their absence" (EWD340). I would phrase the attribution version as: a no counts only when the search that produced it could have said yes. A missing record is a search that could not. Model-based diagnosis already uses "exoneration" for the opposite move, treating a component that was seen to behave correctly as not faulty, and the Exoneration Asymmetry refuses that move unless a kept record supports it. I could not verify the primary paper's wording, so I name the field and do not cite it.
grid-rows: 2
grid-columns: 3
grid-gap: 18
classes: {
free: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
paid: {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
}
q1: "1. controls-ran\nreads: control stream\nyes convicts: harness" {class: free}
q2: "2. required-span-absent\nreads: eviction record\nyes convicts: harness" {class: free}
q3: "3. claimed-and-allowed\nreads: claims, decision trail\nyes convicts: harness" {class: free}
q4: "4. harness-diverges\nreads: replay\nyes convicts: harness" {class: free}
q5: "5. corrected-output-succeeds\nreads: one model call\nyes convicts: model" {class: paid}
out: "A layer clears only when\nevery question that could\nconvict it says no against\na record that was kept" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
The grid above lays the five questions out in run order, with the one paid question last. Order matters, and the chapter enforces it at load time: free questions run before the paid one. Its reason comes from retrieval evaluation, where my earlier rule was to start with context recall because generation metrics are noise when retrieval is broken (Why Your RAG System Cannot Tell When It Is Wrong). A gate's decision cannot be judged if the context it read was assembled wrong, and a model's output cannot be judged if the constraint it broke was never in its window. Questions 1 to 4 are about the harness, so the model question comes last.
That ordering argument is the one the walk itself does not follow.
Why AI agent root cause analysis defaults to the model
Here is the loop at the centre of the chapter's attribute(), from the companion code at tag ch16-attributing-failures. It walks the questions and stops at the first yes:
for question in self.questions: answer, detail = ASKERS[question.id](evidence) findings.append(Finding(question.id, answer, detail)) answers[question.id] = answer if answer is Answer.YES: return Attribution( case=case, layer=question.convicts, by=question.id, findings=tuple(findings), cleared=frozenset(), )An unknown falls through to the next question, which is right for clearing: nothing is cleared on the way down, and clearing is computed at the bottom from the whole answer set. But a yes returns immediately, whatever came before it. When the yes comes from question 5, the walk has convicted the model while four questions about the harness are open.
Question 5 makes this worse, because of what it asks. corrected-output-succeeds replays the recorded context with a correct model output in the seam and asks whether the outcome comes out right. The chapter supplies that answer by hand: yes for inc-0002 and inc-0003, no for inc-0004, and nothing for inc-0005. inc-0004 is the interaction case below, where a correct output still reaches the wrong outcome. A yes shows that the model's output was a cause. It cannot show that it was the only one. A gate that should have refused the wrong output is also a cause, and the counterfactual cannot see it because it never sends the wrong output through. Without the upstream answers, question 5 is the elimination step from the last section with a price attached.
One test in the book's suite asserts this behaviour on purpose. This is the test, verbatim from tests/test_attribution.py at the tag:
def test_replay_agreement_does_not_clear_the_harness( protocol: AttributionProtocol,) -> None: """The sentence this chapter sharpens: agreement is consistency, not correctness.""" result = protocol.attribute("no-divergence", _complete( evictions_kept=False, # one harness question cannot be answered corrected_outcome_right=True, )) assert result.layer is Layer.MODEL findings = {f.question: f.answer for f in result.findings} assert findings["harness-diverges"] is Answer.NO assert Layer.HARNESS not in result.clearedReplay is the test's subject, and its last assertion is correct: the harness is not cleared. But the line above it pins a model conviction on a run where required-span-absent could not be answered, which is the question whose own why in the policy file says the model "cannot be wrong about a constraint it was never shown." The suite has 386 tests at this tag and all of them pass, this one included.
grid-rows: 2
grid-columns: 3
grid-gap: 18
classes: {
open: {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
}
u1: "1. controls-ran\nunknown: falls through" {class: open}
u2: "2. required-span-absent\nunknown: falls through\n(was the constraint\nin the window?)" {class: open}
u3: "3. claimed-and-allowed\nunknown: falls through" {class: open}
u4: "4. harness-diverges\nunknown: falls through" {class: open}
y5: "5. corrected-output-succeeds\nyes" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
v: "Book walk: model\n(returns at the first yes)" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
Box 2 is the one that matters. If the constraint was evicted from the window, the harness owns the failure whatever the model did next, and the record that would say so was never kept. Box 6, in red, is reached anyway.
Agent replay: a replay that never ran reads as agreement
Question 4 holds the second defect. On replay agreement the chapter is right: on its own it clears nothing, because a control that was wrong in the past and is wrong the same way today diverges on nothing. But the question cannot tell a replay that agreed from a replay that never ran:
def _harness_diverges(evidence: Evidence) -> tuple[Answer, str]: if not evidence.signals: return Answer.UNKNOWN, "nothing recorded to replay against" if evidence.divergences: first = evidence.divergences[0] return ( Answer.YES, f"{first.component}: then={first.then} now={first.now}", ) return Answer.NO, "today's controls decide identically, ..."The last detail string is shortened here; it continues "which is not the same as correctly". divergences defaults to an empty tuple, and any run with signals answers no. Nowhere in the chapter's own example is replay() called, for any incident. Its printed lines for inc-0003 and inc-0004, "today's controls decide identically", come from a replay that did not happen. That is the exact shape the chapter warns about for evictions, where evictions == () is true both when nothing was dropped and when nobody kept the record. Evictions got an evictions_kept bit. Replay did not.
This series made the original mistake first. Part 6 built harness replay and wrote: "Unchanged says today's harness would do what September's did, which is the answer that closes most disputes." Its first half is true and its second half is wrong. An unchanged replay shows the harness is consistent with itself. It says nothing about whether the harness was right, and so it closes nothing. Part 6 did get one thing right that the book's attribution code drops: its replay report prints "nothing to compare, not 'unchanged'" for an empty corpus. That distinction is lost the moment a bare tuple of divergences is passed into attribution.
To see how much rests on it, I ran a real replay. I rebuilt today's harness, sent the recorded model output through it again, and compared each control's outcome with what it recorded at the time. For inc-0003, the plausible-but-wrong refund that Part 15 left in the Uncaught Set on purpose, today's harness agrees on every control. That agreement is expected: inc-0003 was recorded on today's harness, so this replay shows only that the code path runs, and the chapter's verdict rests on the other questions. inc-0002 needs two sentences of book history. It is a 480.00 credit allowed by the gate policy as Part II of the book left it, when a comparator that claimed to catch an ungrounded amount let this one through. Chapter 15 later moved that claim to a new comparator and pointed the gate at its verdict. On inc-0002 today's harness does not agree:
today's harness on inc-0002's recorded output: amount-on-invoice then=passed now=absent policy-gate then=allowed now=refusedtoday's harness on inc-0003's recorded output: 0 divergencesChapter 16 attributes inc-0002 to the harness through question 3, because amount-on-invoice claims the ungrounded amount and that amount was allowed. That claim does not exist in the code at the tag. Chapter 15 moved it to another comparator, and the component's catches is now empty. The example puts the old claim back by hand. As a reconstruction of the version that ran, that is fair, but the chapter lists the catches declarations as computed from the run.
Read the claims off the components instead and the walk goes a different way. Question 3 says no, because no component claims the behaviour. Question 4 says no too, from the replay nobody ran. That leaves question 5, and it convicts the model, even though today's harness refuses that same credit at its policy gate and amount-on-invoice is no longer in the path at all. Question 4 can face three kinds of run, and two of them hand it the same data:
grid-rows: 1
grid-gap: 30
ran: "Ran: differ" {
label.near: top-center
style: {fill: "#FFFFFF"; stroke: "#95A5A6"; font-color: "#2C2C2A"}
r: "divergences = (2 rows)\nanswer: yes" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
}
agreed: "Ran: agree" {
label.near: top-center
style: {fill: "#FFFFFF"; stroke: "#95A5A6"; font-color: "#2C2C2A"}
a: "divergences = ()\nanswer: no" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
}
never: "Never ran" {
label.near: top-center
style: {fill: "#FFFFFF"; stroke: "#95A5A6"; font-color: "#2C2C2A"}
n: "divergences = ()\nbook: no fixed: unknown" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
}
Middle and right cells hold the same value, an empty tuple. Only a separate bit can keep them apart, and the book's Evidence has that bit for evictions and not for replay.
Fixing failure attribution: apply the Exoneration Asymmetry to convicting
Two rules make the fix, and they fit in one file that sits beside the book's package without editing it. Kept adds three bits to the book's Evidence, all defaulting to False for the reason evictions_kept does. ask() turns a question whose record was not kept into unknown, and it asks question 3 of each claimant's own record: a gate's decision, or a comparator's or bound's own signal. attribute() lets a yes convict only when every earlier question that could convict a different layer has answered:
"""The Chapter 16 walk with two fixes, kept outside the book's package.1. Every record a question reads says whether it was kept. Replay, the decision trail and the control stream join the eviction record: an empty tuple is not a clean record.2. A later finding depends on the earlier questions. A conviction waits while any earlier question that could convict another layer is still unknown."""from __future__ import annotationsfrom dataclasses import dataclassfrom harness.attribution import ( ASKERS, Answer, AttributionProtocol, Evidence, Finding, Layer,)from harness.roles import Role@dataclass(frozen=True)class Kept(Evidence): replayed: bool = False # replay against today's controls actually ran decisions_kept: bool = False # the run's gate decision trail was stored signals_complete: bool = False # the stream counted no drops for this rundef claimed_and_allowed(ev: Kept) -> tuple[Answer, str]: """Read each claimant's own record: a gate's decision, else its signal.""" if not ev.classified or ev.behaviour is None or ev.claims is None: return ASKERS["claimed-and-allowed"](ev) claimants = sorted(n for n, c in ev.claims.items() if ev.behaviour in c) if not claimants: return ASKERS["claimed-and-allowed"](ev) signal = {s.component: s for s in ev.signals if s.component} allowed, unread = [], [] for name in claimants: s = signal.get(name) if s is None or (s.role is Role.GATE and not ev.decisions_kept): unread.append(name) elif s.role is Role.GATE: if not any(d.gate == name and d.disposition != "allow" for d in ev.decisions): allowed.append(name) elif s.outcome not in ("failed", "exceeded"): allowed.append(name) behaviour = ev.behaviour.value if allowed: return Answer.YES, f"claims {behaviour} and allowed: {', '.join(allowed)}" if unread: return Answer.UNKNOWN, f"no record of what {', '.join(unread)} decided" return Answer.NO, f"{', '.join(claimants)} claimed it and refused"def ask(question_id: str, ev: Kept) -> tuple[Answer, str]: if question_id == "controls-ran" and ev.expected and not ev.signals_complete: fired = {s.component for s in ev.signals if s.component} if ev.expected - fired: return Answer.UNKNOWN, "a control left no signal; drops were not counted" if question_id == "harness-diverges" and not ev.replayed: return Answer.UNKNOWN, "replay was never run for this run" if question_id == "claimed-and-allowed": return claimed_and_allowed(ev) return ASKERS[question_id](ev)@dataclass(frozen=True)class Walk: case: str layer: Layer by: str | None findings: tuple[Finding, ...] cleared: frozenset[Layer] waiting_on: tuple[str, ...] = () def report(self) -> str: lines = [f"{self.case}: {self.layer.value}"] if self.by: lines.append(f" decided by {self.by}") for f in self.findings: lines.append(f" {f.answer.value:<7} {f.question:<26} {f.detail}") if self.waiting_on: lines.append(f" no conviction while unknown: {', '.join(self.waiting_on)}") if self.cleared: names = ", ".join(sorted(layer.value for layer in self.cleared)) lines.append(f" cleared: {names}") return "\n".join(lines)def attribute(protocol: AttributionProtocol, case: str, ev: Kept) -> Walk: findings: list[Finding] = [] answers: dict[str, Answer] = {} for i, question in enumerate(protocol.questions): answer, detail = ask(question.id, ev) findings.append(Finding(question.id, answer, detail)) answers[question.id] = answer if answer is not Answer.YES: continue waiting = tuple( q.id for q in protocol.questions[:i] if q.convicts is not question.convicts and answers[q.id] is Answer.UNKNOWN ) # Stops at the first yes, waiting or not. Safe while the only question # about another layer is last, as in the book's file. layer = Layer.UNATTRIBUTED if waiting else question.convicts by = None if waiting else question.id return Walk(case, layer, by, tuple(findings), frozenset(), waiting) cleared = frozenset( layer for layer in (Layer.HARNESS, Layer.MODEL) if all(answers[q.id] is Answer.NO for q in protocol.questions if q.convicts is layer) ) both = cleared == {Layer.HARNESS, Layer.MODEL} layer = Layer.INTERACTION if both else Layer.UNATTRIBUTED return Walk(case, layer, None, tuple(findings), cleared)This block applies only across layers. An unknown harness question does not stop a later harness question from convicting the harness, because both name the same owner. A harness yes is a finding about the harness whatever else is open. What the block stops is a conviction that moves blame past an open question about a different owner.
On the run that kept nothing, the fixed walk now ends where the evidence does. kept_nothing.py prints both walks for this run. This is the second, with three long lines wrapped:
inc-0005b: unattributed unknown controls-ran no list of controls the deployed harness should run unknown required-span-absent no eviction record was kept for this run unknown claimed-and-allowed the output was never classified against the repertoire unknown harness-diverges replay was never run for this run yes corrected-output-succeeds a correct output through the same harness reached a correct outcome no conviction while unknown: controls-ran, required-span-absent, claimed-and-allowed, harness-divergesThe last line names the four records that would have decided the case, in the order the protocol needs them. A yes from the paid question stays in the findings, so in production that model call is not wasted: once those records exist, the same answer convicts.
That paid answer needs one caution. In production a model proposes the corrected output, and something then has to decide whether the corrected run reached a correct outcome. If a model makes that call too, its verdict forfeits appealable, which the Oracle Deficit section below says may not settle which team owns a defect. So a question-5 yes should convict only once the outcome check on the corrected run is deterministic or reviewed by a person. Until then a model-judged yes may order the queue, and it may not close the incident.
I then ran both walks over the chapter's own incidents with a comparison script, compare.py, which imports the chapter's example helpers and the replay described above. Each incident is recorded through the harness with a signal recorder attached, and each walk runs with and without the missing records. The chapter records inc-0003 and inc-0004 as two runs of the same shape, the same hostile output through the same harness, so the script records it once and reuses that recording for both. Two rows come from a run of its own, where the merge gate refuses a copied argument. After the two replay lines shown earlier, it prints this table:
case book walk fixed walk kept nothing, paid for the counterfactual model unattributed controls list kept, no control stream harness unattributed inc-0002 as printed (claim restored by hand) harness harness inc-0002, today's claims (wrong version) model unattributed inc-0002, same, replay run harness harness merge gate refused, trail kept unattributed unattributed merge gate refused, trail lost harness unattributed inc-0003 as printed (replay never run) model unattributed inc-0003, replay run model model inc-0004 as printed (replay never run) interaction unattributed inc-0004, replay run interaction interactionTwo things in that table carry the argument. On these runs the fixed walk agrees with the book wherever the records exist: with replay run and the trail kept, inc-0002 is still the harness, inc-0003 is still the model, and inc-0004 is still an interaction. Every row where the two walks differ is a row with a record missing, and in each one the book's walk produced a confident layer while the fixed walk names what to go and get. The "today's claims" row is also a version mismatch: it reads claims off today's components for a run on the Part II assembly. The book's hand restoration is the faithful reconstruction there, and the row shows that a claims record needs a version as well. Running the replay is the price of the fix, and the chapter's policy file already prices replay as free.
A test file beside it pins the behaviour against the book's own fixture. Its first test is the book's replay test with the same evidence, asserting that the answer is now unattributed, waiting on required-span-absent. Four more check that a fully recorded run still convicts the model, that a replay nobody ran is unknown, that the kept-nothing run waits on four questions, and that a lost signal is unknown unless the stream counted its drops. All five pass at the tag, and the book's 386 tests still pass beside them, 391 in all. The grid below is the walk at each yes: step 1 is ask(), step 2 is the waiting tuple, and step 3 is question.convicts in the listing above.
grid-rows: 3
grid-columns: 2
grid-gap: 24
q1: "1. A question answers yes" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
a1: "Was its record kept?\nIf not, it is unknown" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
q2: "2. Any earlier question about\nanother layer still unknown?" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
a2: "If yes: unattributed,\nname the missing records" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
q3: "3. Otherwise" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
a3: "Convict the layer\nthis question names" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
q1 -> q2
q2 -> q3
When a missing record may convict a layer
Research and review for this part turned up two more questions that answer from a missing record.
controls-ran answers yes, convicting the harness, when a control the deployed harness should run left no signal. In the book's walk a control that never ran and a signal lost on the way to the store read the same. They need not. Part 6 drops and counts records in copilot mode during a store outage, and its checklist sends runs with dropped records back as inconclusive, not as bypasses. So the fix adds signals_complete. When the stream counted its drops and recorded none for the run, a missing signal still convicts the harness, because emitting it is the harness's job. Without that record it is unknown. Give the opening run its deployed-controls list and still no control stream, and the book's walk convicts the harness on no run records at all, while the fixed walk returns unattributed. One more caveat concerns expected itself. The chapter's example lists every control, so on a refused run the controls after the refusal look as if they never ran. For the two merge-gate rows I set it to the path the run took.
claimed-and-allowed has the same problem in a narrower form. The book builds its set of refusing controls from the decision trail, the per-run record of gate decisions that the book starts in Chapter 3. That trail holds gates only. So a comparator that claims a behaviour can never count as refusing, and a gate whose trail was lost always looks as if it allowed. The merge-gate rows show the second case. The gate refused a copied argument, the trail was lost, and the book's walk reports claims copied-argument and allowed: merge-gate. A trail lives in a store, and a store can lose it without any control misbehaving. So the fix asks each claimant's own record: a gate's decision when decisions_kept is set, or a comparator's or bound's own signal. A missing record is unknown.
grid-rows: 3
grid-columns: 3
grid-gap: 12
h1: "Record" {style: {fill: "#95A5A6"; stroke: "#6F7F80"; font-color: "#2C2C2A"}}
h2: "If it is missing" {style: {fill: "#95A5A6"; stroke: "#6F7F80"; font-color: "#2C2C2A"}}
h3: "Why" {style: {fill: "#95A5A6"; stroke: "#6F7F80"; font-color: "#2C2C2A"}}
c1: "Control signal\n(controls-ran)" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
c2: "unknown, unless the\nstream counted its drops" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
c3: "with drops counted and\nnone lost, it convicts" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
d1: "Gate decision trail\n(claimed-and-allowed)" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
d2: "unknown" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
d3: "a store can lose it\nwith no control at fault" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
Its middle column is the decision: neither record convicts on its absence alone. The chapter leaves both cases unstated.
When no single component failed: interaction failures and the Scope-Blind Guarantee
When both layers clear on kept records and the outcome is still wrong, the case is an interaction. Chapter 16 gives that bucket a signature, using a term from my LangGraph series: a Scope-Blind Guarantee is a runtime guarantee that is real and correctly enforced, "whose unit of scope differs from the unit the adopting engineer assumes", and which no rendered artifact corrects (LangGraph Names the Guarantee, Not the Unit). That audit closed on the line this bucket is for: "All of it was correct, over a unit nobody wrote down, and the model got the blame."
My book has its own example, and Part 14 measured it. Part 9's gate policy lets a copilot refund run with no reviewer when it falls under the 200.00 automatic tier. Act-within-bounds, the band a supervised copilot run holds, allows no unreviewed irreversible refund at all: its ceiling is 0.00. The chapter reports the gap as 50.00 to 200.00. Part 14 traced that figure to the startup check comparing the tier with the closed-loop ceiling, and corrected the gap to 0.00 to 200.00. The gate's unit is the operating mode as it was when the policy was written. The band's unit is the evidence about the run at admission, and that band holds because a person is in the path. Each file is right about its own unit, and a 180.00 refund falls between them without the person seeing it. Both files are harness files, so this is a defect between two harness components, not between the model and the harness. The interaction bucket, as the protocol computes it, collects that kind too: no question about a single component can see it. That is the strongest reason the bucket needs an owner.
grid-rows: 1
grid-gap: 30
gate: "Gate policy" {
label.near: top-center
grid-columns: 1
grid-gap: 10
style: {fill: "#FFFFFF"; stroke: "#95A5A6"; font-color: "#2C2C2A"}
gu: "unit: operating mode\nwhen the policy was written" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
gr: "copilot: automatic\nbelow 200.00" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
}
run: "Between them" {
label.near: top-center
style: {fill: "#FFFFFF"; stroke: "#95A5A6"; font-color: "#2C2C2A"}
r: "180.00 refund,\nno reviewer consulted" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
}
band: "Authority band" {
label.near: top-center
grid-columns: 1
grid-gap: 10
style: {fill: "#FFFFFF"; stroke: "#95A5A6"; font-color: "#2C2C2A"}
bu: "unit: evidence about\nthe run at admission" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
br: "act-within-bounds:\nunreviewed irreversible 0.00" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
}
Both outer columns are green because both components pass their own tests. That red cell exists only where their units meet, which is why no question about a single component can convict it.
This fix sharpens what the bucket means. In the book's walk, inc-0004 lands on interaction with the harness cleared partly by a replay that never ran. Under the fixed walk it reaches interaction only after the replay runs. An agreement from a replay that actually ran is a kept no: one necessary member of the clearing set, never enough on its own. An interaction verdict is earned, not defaulted, which is the chapter's own position, and the fix makes the code match it.
One more caveat belongs in any account of these incidents. inc-0004 in the code is not the 180.00 case above; it is the 40.00 plausible-and-wrong refund with different supplied values, and the chapter's eval case gives it the gate-versus-band description. inc-0003 and inc-0004 are two runs of the same shape. They differ in two values a person supplied, the classification of the output and the counterfactual's answer, and only one of them moves the verdict. No control claims plausible-and-wrong, so question 3 answers no under either classification. Flip corrected_outcome_right and model becomes interaction. The chapter is right that the verdict turns on one boolean, and that boolean is the paid question's answer.
The Oracle Deficit: what an LLM judge's verdict may decide
Question 5 needs a verdict on what a correct output would have been, and so does every eval case built from an incident. Who may give that verdict is the subject of the chapter's second named concept. Part 1 of this series argued that published harness wins were scored by an oracle their domain supplied. My book lists what that test-suite oracle gave for nothing: a verdict that is free, immediate and unambiguous, and that covers every run. The Oracle Deficit is the property a substitute scorer gives up. In the chapter's words, "which one it forfeits is what the score may not be used for." The failure it names is a good scorer quoted in a decision its deficit does not cover.
Its table has a flaw that shows once you read it against the chapter's own prose. "Total" is defined as covering every run, not a sample. In the table the deterministic scorer forfeits it. But a deterministic check does run on every run. What it lacks is coverage of every way an output can be wrong: it can say an amount appears on no invoice, and it cannot say the customer wanted an explanation, not a refund. Human review keeps "total", yet the chapter says a reviewer's ninety seconds per case "is why it will be applied to a sample." Two properties share one name.
"Unambiguous" has a similar problem. Human labels are not unambiguous: the expert annotators behind Who&When report that "The uncertain percentages across different annotators range from 15% to 30%" (Zhang et al.). What a human verdict has, and a judge's does not, is something underneath it to argue with. Chapter 16 already uses that sense when it says of a judge that "there is nothing underneath the score to appeal to." Mughal and colleagues put the same test to LLM oracles in general: "The first question to ask of any LLM oracle is therefore what one would point to in defending its verdict" (Mughal et al., 2026). I call that property appealable: a verdict with something under it that a person can argue with. The Oracle Deficit is also distinct from what Molina and Gorla call oracle deficiencies, which are defects in the oracles an LLM generates. A scorer can be right on every case it scores and still have a deficit.
So I split "total" into two properties, renamed "unambiguous" to appealable, and recomputed the table with those definitions. The result follows from the choices in two dictionaries, and the choice that moves a cell is that human review forfeits every-run while ranking a backlog needs it. Take every-run off the human row and the chapter's original answer for ranking comes back. Each use takes the half of "total" its job needs. Ranking needs both, because ranking a sample ranks the sample and what a scorer cannot see is off the list, not low on it. A rate needs every failure mode but not every run, because a drawn sample with an interval is enough. Blocking a deploy and settling a dispute need neither half. Here is the core of deficit.py, standard library only:
FORFEITS = { # Runs on every run; sees only what has a mechanical form. Scorer.DETERMINISTIC: {Prop.EVERY_MODE}, # Runs on every run and reads everything; nothing under the score to appeal to. # Its every-mode is nominal: it shares blind spots with the model it judges. Scorer.JUDGE: {Prop.APPEALABLE}, # Sees every mode and can say why; costs minutes per run, so it gets a sample. Scorer.HUMAN: {Prop.FREE, Prop.IMMEDIATE, Prop.EVERY_RUN},}NEEDS = { "block-a-deploy": {Prop.IMMEDIATE, Prop.APPEALABLE}, "rank-a-backlog": {Prop.EVERY_RUN, Prop.EVERY_MODE}, "settle-a-dispute": {Prop.APPEALABLE}, # A drawn sample with an interval is enough; every run is not needed. "report-a-rate": {Prop.EVERY_MODE, Prop.APPEALABLE},} scorer forfeits may not deterministic every-mode rank-a-backlog, report-a-rate judge appealable block-a-deploy, settle-a-dispute, report-a-rate human every-run, free, immediate block-a-deploy, rank-a-backlog block-a-deploy deterministic rank-a-backlog judge settle-a-dispute deterministic, human report-a-rate humanOne cell moves against the chapter's table. Human review can no longer rank a backlog, because ranking a sample ranks the sample, which is the chapter's own comment on that use. With these choices, a deterministic check is the only scorer that may block a deploy automatically, and human review the only one that may report a rate. The judge's ranking cell is the weakest in the table. It holds only if you grant the judge every failure mode, and its coverage there is nominal. Take every-mode away from the judge too, and no scorer may rank a backlog, which says ranking needs a judge plus randomly drawn cases. The rejection rate Part 7 used for its disclosure test belongs in the deterministic row. It is a comparator's verdicts over a list of likely wrong answers. It sees only the failure modes on that list, so it may not stand in for the failure rate. The grid below puts each scorer's loss beside the use that is left to it alone.
grid-rows: 4
grid-columns: 3
grid-gap: 10
h0: "Scorer" {style: {fill: "#95A5A6"; stroke: "#6F7F80"; font-color: "#2C2C2A"}}
h1: "Gives up" {style: {fill: "#95A5A6"; stroke: "#6F7F80"; font-color: "#2C2C2A"}}
h2: "The one use only it may serve" {style: {fill: "#95A5A6"; stroke: "#6F7F80"; font-color: "#2C2C2A"}}
d0: "Deterministic check" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
d1: "every failure mode" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
d2: "block a deploy" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
j0: "LLM judge" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
j1: "appealable" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
j2: "rank a backlog" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
u0: "Human review" {style: {fill: "#C2185B"; stroke: "#8E1243"; font-color: "#FFFFFF"}}
u1: "free, immediate,\nevery run" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
u2: "report a rate\n(from a drawn sample)" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
A misranking is cheap only for failures the judge can see: an incident ranked too low is looked at later. Failures the judge shares with the model it judges are never ranked at all, so the randomly drawn cases described below are what cover them. And a judge can extend a rate, correctly, if it is calibrated against human labels. Prediction-powered inference gives "provably valid confidence intervals" when a model's predictions supplement a labelled sample (Angelopoulos et al., 2023). That rate still rests on human labels, which is what the human row says.
For attribution, the table gives the rule this part needs. An LLM reading your traces may rank the backlog of incidents. That is close to how LangSmith, Arize and Datadog describe their trace analysis: grouping and summarising failing traces so engineers do not read thousands by hand. Datadog's own post says "Engineers still decide which failure groups represent high-priority issues" (Datadog, 2026). A judge's verdict may not settle which team owns a defect, because that is a dispute, and a dispute needs an appealable verdict. And where the records are missing, the protocol answers unknown while a judge still writes a paragraph.
Track the unattributed rate, not the attributable one
Run the fixed walk over a quarter of incidents and some share will come back unattributed. Report it in two parts: cases waiting on a missing record, and cases waiting only on a counterfactual nobody paid for. The first part measures your records. It falls when you start keeping one and rises when a store is retired or a retention window expires. Driving it to zero is easy and worthless. Treat every missing record as no and every case attributes. With the paid question answered yes, the cases go to the model; left unanswered or answered no, they go to interaction, which is where the chapter says they drift. My book's walk did the first for the model with a single answered question. "Cleared" also means cleared by the questions this protocol asks. The load-time check refuses a layer with no convicting question, and it cannot tell a layer with too few.
Scotland abolished its third verdict, not proven, from 1 January 2026, with its Justice Secretary calling it "a widely misunderstood verdict and one with no statutory definition" (Scottish Government, 2025). Any third value risks the same fate, so unattributed has to be built differently. It names the missing records. It describes the evidence, not the accused. And it has an owner, the team that owns the stores, so it turns into work instead of a stigma. That ownership closes a loop:
grid-rows: 2
grid-columns: 2
grid-gap: 24
kept: "1. Records kept at the time\n(signals, evictions,\ntrail, replay)" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
walk: "2. Fixed walk over\na quarter of incidents" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
owner: "4. Owner of the stores keeps\nthe named record; next\nquarter starts back at 1" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
rate: "3. Unattributed rate,\neach case naming\nits missing records" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
Each unattributed case names a record, the record becomes a work item for whoever owns the store, and next quarter's rate shows whether it was kept.
Three things will skew the number. First, the corpus is biased: cases built from reported incidents miss failures nobody noticed, and a control that has silently stopped working is the class nobody reports. My note on RAG feedback described the same decay, with "Your quality metrics are now based on 10% of traffic" six months in (Closing the Loop). Draw some cases at random from allowed runs, and keep drawn and reported cases apart. Second, a model upgrade expires half of every stored case. A recorded output stays valid, but the counterfactual's answer was measured on a model that no longer serves, which is the dated Plant Model entry from Part 2 arriving in the eval set. Record which model answered question 5, and treat the model half of an attribution as expiring on upgrade. Third, give the interaction bucket an owner before it has cases in it. Two teams can each correctly believe their component is fine, and the bucket will sit untouched.
A record can convict a layer, and only a full set of no answers, each read from a record kept at the time, can clear one. Holding a conviction while an earlier question about another layer is still unknown is that same rule, applied to convicting.
Checklist: attributing an agent failure to a layer
- Every attribution question answers yes, no or unknown, and
unknownnames the record that was not kept. - Every record a question reads has a "was it kept" bit, including replay, the decision trail and the control stream, not only evictions. An empty tuple is never read as a clean record unless its kept bit is set.
- A conviction waits while any earlier question that could convict a different layer is
unknown: the Exoneration Asymmetry applied to convicting. Test it with your own copy of the book's replay test: no eviction record plus a paid counterfactual must not return the model. - A missing control signal convicts the harness only when the stream counted its drops and recorded none for the run. Otherwise it is
unknown. - A question-5
yesconvicts the model only when the outcome of the corrected run was checked deterministically or by a person. A model-judgedyesorders the queue and does not close the incident. - Replay is actually run, by sending the recorded output through today's harness. "No divergence" is reported only for a replay that happened.
- A layer is cleared only when every question that could convict it said
no. A layer with no convicting question is refused at load, as the chapter does. - A conviction is reported as "a layer that owns a defect", not "the root cause". The walk stops at the first finding, and failures often have several contributors: Cook makes the general case in "How Complex Systems Fail", and MP-Bench measures it for multi-agent systems.
- An LLM judge over traces ranks the incident backlog. It does not assign ownership, block a deploy or produce the failure rate.
- The failure rate comes from human labels on a drawn sample, with its interval. A judge may extend it only when calibrated against those labels.
- The unattributed rate is tracked per quarter, with each case's missing records routed to whoever owns the store.
- Every stored counterfactual records the model that answered it, and expires on upgrade.
- If you can start keeping only one record this quarter, follow question order: the control stream with the list of controls that should have run, then the eviction record, then the decision trail, then replay. A model conviction waits on all four, so keep them in this order.
Harness Engineering book, Chapter 16: Attributing Failures to a Layer
This is Part 16 of a series that follows Harness Engineering for Production AI Systems one chapter at a time. Chapter 16, "Attributing Failures to a Layer," builds the attribution protocol as a policy file with three load-time checks, fixes a signal schema whose required reason field held a label for ten chapters, ships four incidents as eval cases with required provenance, and covers why a model should not be asked which layer failed. This part adds the upstream block and the replay and decision-trail bits, and corrects the chapter in four places. Its walk convicts across open questions, and the chapter's own replay test pins it. Replay never runs in its example. inc-0002's claim is restored by hand while the prose calls it computed. And its Oracle Deficit table uses "total" for two properties. Code for the chapter is in the ch16-attributing-failures tag of the companion repository.
References
- Kumar, R. (2026). Harness Engineering for Production AI Systems, Chapter 16, "Attributing Failures to a Layer". https://ranjankumar.in/harness-engineering-for-production-ai-systems
- Kumar, R. (2026). harness-engineering-code, tag
ch16-attributing-failures. https://github.com/ranjankumar-gh/harness-engineering-code/tree/ch16-attributing-failures - Kumar, R. (2026). The Oracle Problem: Why Most AI Agent Loops Fail. https://ranjankumar.in/loop-engineering-agent-control-loop
- Kumar, R. (2026). LangGraph Names the Guarantee, Not the Unit. https://ranjankumar.in/langgraph-scope-blind-guarantees-runtime-audit
- Kumar, R. (2026). Why Your RAG System Cannot Tell When It Is Wrong. https://ranjankumar.in/rag-engineering-evaluation-evals-blind-spot
- Kumar, R. (2026). Closing the Loop: How to Actually Measure RAG Quality in Production. https://ranjankumar.in/closing-the-loop-how-to-actually-measure-rag-quality-in-production
- Kumar, R. (2026). Harness Engineering for Production AI Systems series, Parts 1, 2, 5, 6, 7, 9, 14 and 15. https://ranjankumar.in/guides/harness-engineering-book
- Zhang, S., Yin, M., et al. (2025). Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems. ICML 2025. https://arxiv.org/abs/2505.00212
- Zhang, G., et al. (2025). AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems? ICLR 2026. https://arxiv.org/abs/2509.03312
- Ahasanuzzaman, M., et al. (2026). FALAT: Tracing Failures in LLM Agent Trajectories via Dependency-Guided Search. https://arxiv.org/abs/2606.00765
- In, Y., et al. (2026). Rethinking Failure Attribution in Multi-Agent Systems: A Multi-Perspective Benchmark and Evaluation. https://arxiv.org/abs/2603.25001
- Microsoft Research (2026). Systematic debugging for AI agents: Introducing the AgentRx framework. https://www.microsoft.com/en-us/research/blog/systematic-debugging-for-ai-agents-introducing-the-agentrx-framework/
- Dijkstra, E. W. (1972). The Humble Programmer (EWD340). https://www.cs.utexas.edu/~EWD/transcriptions/EWD03xx/EWD340.html
- Mughal, A. H., et al. (2026). LLM-Based Test Oracles: Source-of-Authority Taxonomy - A Systematic Literature Review. https://arxiv.org/abs/2607.05031
- Molina, F., & Gorla, A. (2024). Test Oracle Automation in the era of LLMs. https://arxiv.org/abs/2405.12766
- Angelopoulos, A. N., Bates, S., Fannjiang, C., Jordan, M. I., & Zrnic, T. (2023). Prediction-Powered Inference. Science. https://arxiv.org/abs/2301.09633
- Cook, R. I. (1998). How Complex Systems Fail. https://how.complexsystems.fail/
- LangChain. LangSmith Insights. https://docs.langchain.com/langsmith/insights
- Arize. Arize AX: Explore your traces. https://arize.com/docs/ax/observe/agent-insights
- Pan, T. (2026, April 19). The Vanishing Blame Problem in AI Incident Post-Mortems. https://tianpan.co/blog/2026-04-19-vanishing-blame-ai-incident-postmortems
- Pan, T. (2026, June 1). The Postmortem Template With No Row for the Model's Inference. https://tianpan.co/blog/2026/06/01/the-postmortem-template-with-no-row-for-the-models-inference
- Datadog (2026, June 9). Improve AI agent quality with Bits Evals. https://www.datadoghq.com/blog/bits-evals/
- Scottish Government (2025). Abolition of not proven verdict. https://www.gov.scot/news/abolition-of-not-proven-verdict/
Related Articles
- Deterministic Constraint Systems: Building Tool Registries That Keep Agents in Scope
- AI Agent Approval Gates: Failed Checks Must Never Widen
- Testing AI Agent Guardrails: Green Suites Hide Dead Controls
- Strict Tool Schemas Aren't Enough: Narrow LLM Tool Calls






