This is a constructed case, from the billing support agent this series and its book are built around. The question came from legal in March 2026, about a run the previous September. A customer said our billing agent had told them a refund was approved and then never paid it. Somebody had to say what the system did on the night of 10 September 2025, and why. This part builds the AI agent decision log that would have answered legal, and the check that catches an action no control ever decided.
Our tracing backend still had the run. Six tool calls, 31,000 tokens, eleven seconds end to end, and an escalate_to_human call at 02:14:07. Every one of those fields was correct. None of them said why the run escalated instead of refunding.
For about a millisecond, the reason existed. A comparator had found that the 940.00 the model proposed matched no invoice row, and the refund gate had read that verdict and refused. Both facts were local variables in functions that returned normally. Whoever answered legal had to check out the commit the deploy log said was live that night and read the gate's code. It took most of a day. What it produced was an argument about what the code would have done, and nobody could check it.
My claim has two steps, and the first is familiar. Agent tracing records what the model saw and did, while the question that arrives months later is what the harness's own controls decided and why. For controls you write yourself, a trace holds no such record unless you emit one. So the harness needs a decision log of its own. I call it the Second Stream: a harness-owned record of control facts, one per control per decision, kept on its own retention. It is designed backwards from what I call the Six-Month Question: why did the system do that, asked half a year later by somebody who was not there, about a run nobody remembers, on a harness version that no longer exists.
Step two is what a decision log alone does not give you. A decision log records only the checks that ran. A gate that allowed leaves a record. A refund that went down a path with no gate on it leaves none, in a trace or in a decision log. So absence means nothing until the harness declares, for each consequential effect, which decisions that effect is owed. I call that declared set its owed decisions. With it, a missing or non-allowing decision is a finding you can page on.
Decision logs themselves are old. Open Policy Agent (OPA) can log every policy decision, allows included, with the bundle revision that made it (OPA decision logs). Kubernetes writes authorization.k8s.io/decision and a free-text authorization.k8s.io/reason into its audit events (Kubernetes audit annotations). Among AI-specific writing, the closest statement of the problem is Rick Hightower's: a failure went unnoticed because "the harness recorded nothing about its own decisions" (Hightower, 2026). His fix keeps those decisions as spans in the same trace, so by default they live on trace retention and sampling, and it records raw tool results.
This check is old too. Saltzer and Schroeder's principle of complete mediation says "Every access to every object must be checked for authority" (Saltzer and Schroeder, 1975). Finance teams apply the same idea when they look for payments with no matching approval, and so do security teams that alert on an action with no authorisation before it. Two things change when the actor is a model, and they are this part's contribution.
First, the model picks the tool, and the tool set can change after CI ran, as tools are discovered from MCP servers or added as plugins. A registration keyed by tool name misses the same refund reached through apply_credit. So owed decisions have to be declared per effect, and a tool with no declared effect is a finding at run time, not only a lint in CI. Second, the check reads absence, so it is only as sound as the log is complete. Sampling, a store outage and a late action record can each turn an honest pass into an apparent bypass. That is why the sampling and outage policies are written against the log's completeness, with cost second.
Why agent traces can't replace an AI agent decision log
A trace records run facts: latency, tokens, cost, which tool ran, what it returned, whether the request succeeded. You page on these. A control records control facts: this comparator failed, this gate refused for this reason, this bound had 1,060.00 of its 2,000.00 daily ceiling used, this sensor removed 204 code points. These are different kinds of fact, and the second cannot be derived from the first.
A run fact records an outcome and a control fact records a decision. Tracing has other blind spots for agents, such as exit-time metrics that never fire for a loop that never exits (AI control plane observability). This one is different. When a decision comes out the same way as doing nothing, the outcome stream has no trace of it. An allowed refund and a refund that went down a path with no gate on it both end in issue_refund succeeding.
Both streams sit side by side for one run in the diagram below. Read across a row. The left column is what your tracing backend will still have in March, and the right column is what legal will ask about.
grid-rows: 1
grid-gap: 40
run: "Run facts (the trace)" {
label.near: top-center
grid-columns: 1
grid-gap: 14
style: {fill: "#EAF2FB"; stroke: "#2C6FB0"; font-color: "#2C2C2A"}
r1: "6 tool calls, 31,000 tokens, 11 s" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
r2: "issue_refund proposed: 940.00" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
r3: "escalate_to_human at 02:14:07" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
r4: "status: OK" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
}
ctl: "Control facts (the Second Stream)" {
label.near: top-center
grid-columns: 1
grid-gap: 14
style: {fill: "#EFF9F1"; stroke: "#3F9E52"; font-color: "#2C2C2A"}
c1: "daily-refund-bound: within\n1,060.00 of 2,000.00 refunded today" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
c2: "amount-on-invoice: failed\n940.00 matches no invoice row" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
c3: "refund-gate: refused\nescalated: amount-on-invoice failed" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
c4: "harness 0.6.0, acme/reasoner-3.1\non every signal" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
}
Nothing in the left column is wrong, and no amount of it produces the right column. Look at the bottom-right cell as closely as the red one. Without the harness version on the decision itself, "which rules were in force that night" is a question about deploy history, not about the run.
What framework guardrail spans record, and what they leave out
Some frameworks do record guardrail runs, and the table shows what each one keeps. The OpenAI Agents SDK runner opens a guardrail_span() on every guardrail run and exports {"type": "guardrail", "name": ..., "triggered": ...}, so a pass shows up as triggered: False (span_data.py, run_internal/guardrails.py). The other rows come from the Langfuse observation types and versioning pages, the OpenInference conventions and the ApplyGuardrail reference.
| Record | Pass recorded | Reason | Version on the decision | Covers your business controls |
|---|---|---|---|---|
OpenAI Agents SDK guardrail_span | yes, triggered: False | no | no | only SDK guardrails |
Langfuse guardrail observation | if you send it | free-form input/output | optional version per observation | if you instrument it |
Bedrock ApplyGuardrail, FULL scope | yes | actionReason when it intervenes | yes, guardrailVersion | only Bedrock guardrail policies |
OpenInference GUARDRAIL span kind | if instrumented | no field for it | no field for it | framed as content safety |
| OpenTelemetry GenAI conventions | no guardrail convention exists | - | - | - |
| A refund comparator written as plain code | nothing, unless you emit it | - | - | - |
The gap is widest where the consequential decisions live. A content-safety filter gets a span. The comparator that checks 940.00 against the ledger is a Python function. It gets nothing unless you emit it yourself, as a custom span or attribute that nobody standardises. Rewritten as an SDK guardrail, it gets a name and a boolean. Span-based records are also kept on trace terms. Under head sampling in the Langfuse SDK or an OpenTelemetry SDK they are sampled per trace and retained as long as the trace backend keeps things, and among the span-based records, only Langfuse's free-form payload has a place for a reason. Bedrock fills actionReason only when it intervenes, and OpenTelemetry's gen_ai.evaluation.result has an explanation field for quality scores, not for permissions. Bedrock's assessment is an API response, so it is kept only if you store it yourself. As of November 2026, the OpenTelemetry GenAI conventions are still at development status, with no release. They define a gen_ai.evaluation.result event for output quality and nothing for a permission decision (GenAI events registry).
The wrong way: log the refusals and the context
When the March question goes unanswered, the instinct is to log more. Two versions of "more" are common. One logs only the interesting cases, because successful refunds are the volume and refusals are the story. The other logs the context the model saw, so an investigator can reconstruct the reasoning. I argued the second one on this blog in February: "[...] context can be megabytes per decision. But it's necessary for agent systems where decisions are non-deterministic [...]" (Agent Audit Trails). That post logged the actions as well, and in the demo below the action records would be identical too. I now think it pointed the log at the model when the question was about the controls, and this part narrows it.
Below, both versions are cut down to a few lines. The gate logs a warning with the ticket text when it refuses and stays silent when it allows. Its refund tool logs the full context the model saw, which was the February fix. A second dispatch path skips the gate entirely. It is the kind of second caller that Part 3's one-call-site rule exists to catch.
# wrong_way.pyimport logginglog = logging.getLogger("billing")records: list[str] = []class Keep(logging.Handler): def emit(self, record: logging.LogRecord) -> None: records.append(record.getMessage())log.addHandler(Keep())def refund_gate(amount: float, on_invoice: bool, ticket_body: str) -> str: if on_invoice: return "allowed" # happy path: nothing worth logging log.warning("refund %.2f refused, ticket: %s", amount, ticket_body) return "refused"RULES = "You are the billing support assistant. Refunds need a matching row. "SYSTEM_PROMPT = RULES * 40context_log: list[str] = []def issue_refund(amount: float, ticket_body: str) -> str: context_log.append(SYSTEM_PROMPT + ticket_body) # "log the context" (February) return f"refunded {amount:.2f}"def handle(amount: float, on_invoice: bool, ticket_body: str, fast_path: bool) -> str: if fast_path: # a second dispatch path return issue_refund(amount, ticket_body) if refund_gate(amount, on_invoice, ticket_body) == "allowed": return issue_refund(amount, ticket_body) return "escalated"for run, fast_path in [("r_a gate allowed", False), ("r_b gate never called", True)]: before = len(records) result = handle(120.00, True, "Please refund the duplicate SSO seat.", fast_path) print(f"{run:<22} {result:<16} control records: {len(records) - before}" f" context logged: {len(context_log[-1]):,} chars")print(f"context records identical: {context_log[0] == context_log[1]}")handle(940.00, False, "Please refund the duplicate SSO seat.", fast_path=False)print(f"refusal record: {records[-1]}")r_a gate allowed refunded 120.00 control records: 0 context logged: 2,757 charsr_b gate never called refunded 120.00 control records: 0 context logged: 2,757 charscontext records identical: Truerefusal record: refund 940.00 refused, ticket: Please refund the duplicate SSO seat.Both runs refunded 120.00, and both left zero control records. In the first the gate decided; in the second there was no gate. Each run added 2,757 characters to the context log, and the two records are identical, because the gate's decision was never in the context. A trace would show the same issue_refund call in both. This diagram marks where the information was lost.
direction: down
classes: {
run: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
ctl: {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
bad: {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
}
a: "r_a: gate called" {class: run}
b: "r_b: fast path" {class: run}
gate: "refund_gate: allowed\n(logs nothing)" {class: ctl}
skip: "no gate on this path" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
refund: "issue_refund(120.00)" {class: run}
log: "Control log for both runs: empty\nsame trace, same outcome" {class: bad}
a -> gate
b -> skip
gate -> refund
skip -> refund
refund -> log
Both paths end at the same issue_refund and the same empty log. The refusal branch has its own problem, and the last line of the output shows it. It writes the ticket body into a log store that is kept far longer than the ticket. The February post saw this too: it put full content in a separate context store and suggested "content hashes instead of full content for sensitive sources." What it still did was point the audit trail at the model's context, on the theory that the "why" lives there. For the decisions a harness makes, it does not. For the 02:14 refusal, the why is two short facts, and neither is in the window.
What fields an agent decision log needs: the Six-Month Question
Schemas designed forwards produce fields. A schema designed backwards from a question produces answers, so start from the questions that actually get asked about a past run. I find four, and each one forces a field.
grid-rows: 4
grid-columns: 2
grid-gap: 22
q1: "1. Why was this allowed or refused?" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
a1: "component + role + outcome + reason\non every control decision" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
q2: "2. What did the model ask for?" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
a2: "a PROPOSED signal, kept apart\nfrom what was ACTED" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
q3: "3. Which controls were in force?" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
a3: "harness_version + model_id\non every signal, not on the run" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
q4: "4. Has this control ever done anything?" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
a4: "a count for every outcome,\nthe boring ones included" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
q1 -> q2
q2 -> q3
q3 -> q4
Question 1 is the March question, and the most common form of the Six-Month Question. It needs the decision and its reason. Reviewers cut the reason first, because free text cannot be aggregated reliably. Keep it anyway. It is the field a human reads first, and a refusal without one is the day of reading commits this part opened with.
Question 2 is what the dispute is usually about. The customer is arguing about the 940.00 the model proposed, not about the escalation that happened instead, so the proposal is its own signal.
Question 3 is why the version goes on every signal. An obvious design puts harness_version and model_id on a run-started event. It breaks the first time a run spans a model fallback, a config reload or, in a long-running agent, a deploy. Your Plant Model measurements are also keyed by the model id, so a decision without one cannot be matched to the rates that were true when it was made.
Question 4 is the one that costs money, and it gets its own section below. Birgitta Böckeler asks the same thing about coding-agent harnesses: "If sensors never fire, is that a sign of high quality or inadequate detection mechanisms?" (Böckeler, martinfowler.com). Boring records answer half of that question: they show the sensor ran. Its other half needs an input seeded to make it fire, like the daily canaries Part 4 prescribed for rare-event steps. Without the boring records you cannot even ask.
The role field from the Role Test makes all four answerable by someone who does not know your component names. "Every gate refusal in September" is a query a new engineer can write, and it survives the day refund-gate is renamed.
Validating agent decision log signals at emit time
My book's companion code for this chapter has the right fields. When I probed it at its ch06-signals tag, it let through more than the chapter says. A ticket body passed the never-raw check as long as it started with sha256:. Raw text under any key not on the list passed. A blank reason passed. A schema version was loaded and never written onto a signal. Those are the cases the schema below refuses. Types come first. Role is Part 3's enum without the residual, because a residual is application code and emits no control facts.
# second_stream.pyfrom __future__ import annotationsimport hashlibimport hmacimport jsonimport randomfrom collections import Counterfrom decimal import Decimalfrom dataclasses import asdict, dataclass, field, replacefrom datetime import datetime, timedelta, timezonefrom enum import Enumfrom typing import Callable, Iterable, Mappingclass Role(str, Enum): SENSOR = "sensor" COMPARATOR = "comparator" GATE = "gate" BOUND = "bound"class Kind(str, Enum): OBSERVED = "observed" # a sensor changed or dropped something PROPOSED = "proposed" # the model asked for an action CONTROL = "control" # a comparator, gate or bound decided ACTED = "acted" # a tool ranclass Outcome(str, Enum): OBSERVED = "observed" PASSED = "passed" FAILED = "failed" ALLOWED = "allowed" REFUSED = "refused" WITHIN = "within" EXCEEDED = "exceeded" RAISED = "raised"BORING = {Outcome.PASSED, Outcome.ALLOWED, Outcome.WITHIN}@dataclass(frozen=True)class Digest: value: strdef digest(text: str, key: bytes) -> Digest: """Keyed, so only the holder of the key can test a guess against it.""" mac = hmac.new(key, text.encode("utf-8"), hashlib.sha256).hexdigest() return Digest("hmac-sha256:" + mac[:32])@dataclass(frozen=True)class Signal: run_id: str at: datetime kind: Kind harness_version: str model_id: str component: str | None = None role: Role | None = None outcome: Outcome | None = None reason: str = "" detail: Mapping[str, str | Digest] = field(default_factory=dict) schema_version: int = 0 # stamped by emit(), never by the callerraised is what the executor records when a control throws. A control that threw is neither working nor failing. It is absent, and a query on raised finds the controls that threw. A handler that swallows the error inside the control and emits allowed is invisible to every check in this part. Only a refusal test or a replay against a known-bad input finds it. Outcome is a closed vocabulary because the readiness query later in this series asks "has this gate ever emitted refused", and that query breaks the first time someone writes "refuse". Digest is a type rather than a string prefix, so a ticket body that happens to start with sha256: no longer passes. Code can still build Digest("raw text") by hand, and that is a one-line pattern a reviewer can grep for. Now the checks:
class SchemaViolation(Exception): pass@dataclass(frozen=True)class SignalSchema: version: int = 2 never_raw: frozenset[str] = frozenset( {"ticket_body", "reply", "kb_article", "account", "invoice_rows"} ) max_value_chars: int = 64 max_detail_chars: int = 500 def check(self, s: Signal) -> None: if not (s.run_id and s.harness_version and s.model_id): raise SchemaViolation("run_id, harness_version and model_id are required") if s.at.tzinfo is None: raise SchemaViolation("at must carry a timezone") if s.kind is Kind.CONTROL: who = s.component or "<unnamed>" if not (s.component and isinstance(s.role, Role)): raise SchemaViolation(f"control signal from {who}: " "component and role required") if not isinstance(s.outcome, Outcome): raise SchemaViolation(f"control signal from {who}: " f"unknown outcome {s.outcome!r}") label = f"{s.role.value} {s.outcome.value}" if not s.reason.strip() or s.reason.strip() == label: raise SchemaViolation(f"control signal from {who}: reason is required") total = 0 for key, value in s.detail.items(): if isinstance(value, Digest): total += len(value.value) continue if key in self.never_raw: raise SchemaViolation(f"{key} may only be recorded as a Digest") if len(value) > self.max_value_chars: limit = self.max_value_chars raise SchemaViolation(f"{key}: {len(value)} chars, limit {limit}") total += len(value) if total > self.max_detail_chars: raise SchemaViolation(f"detail: {total} chars, limit {self.max_detail_chars}")Some of these rules go further than the book's code. The 64-character cap on each plain value stops a paragraph of ticket text from arriving under a key nobody listed. Amounts, counts, labels and step names fit, but a sentence of customer prose does not. A 500-character cap applies to the whole detail, not to each value. And a reason that is exactly the role-and-outcome label, "gate refused", counts as missing, because those two fields are already on every signal and the label answers nothing. The check is an exact match, so "Gate refused" still passes, which is one more reason the reason field is reviewed by a person. The companion code's Chapter 16 docstring records the same lesson: the required-field check "asks whether the field is present, and a label is present."
SignalLog enforces the schema, in a fixed order inside emit.
EFFECTS = {"issue_refund": "money-out", "apply_credit": "money-out", "send_reply": "promise", "lookup_invoice": "read", "escalate_to_human": "handoff"}OWED = {"money-out": ("daily-refund-bound", "amount-on-invoice", "refund-gate"), "promise": ("reply-claims-check",), "read": (), "handoff": ()}# derived, never listed by hand: a second list would drift from the effect mapCONSEQUENTIAL = frozenset(tool for tool, effect in EFFECTS.items() if OWED[effect])class SignalLog: def __init__(self, schema: SignalSchema, boring_sample_rate: float = 1.0, rng: random.Random | None = None, spare_consequential: bool = True) -> None: self.schema = schema self.spare = spare_consequential self.rate = boring_sample_rate self.rng = rng or random.Random(0) self.records: list[Signal] = [] self.counts: Counter[tuple[str, str, str, str]] = Counter() self.keep_all: set[str] = set() # runs that proposed a consequential tool self.dropped: Counter[tuple[str, str]] = Counter() def emit(self, signal: Signal) -> None: self.schema.check(signal) # refuse before anything is kept signal = replace(signal, schema_version=self.schema.version) if (self.spare and signal.kind is Kind.PROPOSED and signal.detail.get("tool") in CONSEQUENTIAL): self.keep_all.add(signal.run_id) if signal.kind is Kind.CONTROL: day = signal.at.date().isoformat() key = (signal.component, signal.outcome.value, signal.harness_version, day) self.counts[key] += 1 sampled_out = self.rng.random() >= self.rate if (signal.outcome in BORING and signal.run_id not in self.keep_all and sampled_out): return # record sampled out, count kept self.records.append(signal) def lose(self, signal: Signal) -> None: """Store down in copilot mode: drop the record, keep its counts.""" self.dropped[(signal.run_id, signal.detail.get("proposal", ""))] += 1 if signal.kind is Kind.CONTROL: day = signal.at.date().isoformat() key = (signal.component, signal.outcome.value, signal.harness_version, day) self.counts[key] += 1 def why(self, run_id: str) -> list[Signal]: return [s for s in self.records if s.run_id == run_id and s.kind is Kind.CONTROL] def decided(self, component: str, outcome: Outcome) -> int: return sum(n for (c, o, _, _), n in self.counts.items() if c == component and o == outcome.value) def repeated_components(self, run_id: str) -> list[str]: seen = Counter((s.detail.get("proposal"), s.component) for s in self.why(run_id)) return [c for (_, c), n in seen.items() if n > 1] def versions_on(self, day: str) -> set[str]: # read from the counts, which are never sampled return {v for (_, _, v, d) in self.counts if d == day}direction: down
sig: "Signal\nfrom a control" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
check: "1. schema.check\nrefuse or pass" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
stamp: "2. stamp\nschema_version" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
count: "3. count\n(component, outcome,\nversion, day)" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
keep: "4. keep record\n(boring ones sampled until the\nrun proposes a consequential tool)" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
err: "SchemaViolation\nin the author's test run" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
sig -> check
check -> stamp: "valid"
stamp -> count
count -> keep
check -> err: "invalid"
Order is the design here. The check runs before anything is stored, so a malformed signal never reaches the warehouse looking like evidence. The count runs before the sampling decision, so dropping a record never drops the fact that a decision happened. Because the count carries the harness version, "has this gate refused anything since 0.6.0 shipped" is answerable from counts alone.
Sampling never touches a run once it has proposed a consequential tool. From the moment a run asks for issue_refund, every record in it is kept, the boring ones included. Records from before the proposal can still be sampled, but none of them can be owed to that refund, and their counts are kept either way. keep_all here lives in memory. In a run that can resume on another pod after a checkpoint, the keep decision has to travel with the run state, or the resumed half gets sampled.
The log also stamps the schema version itself, so a reader in March running schema 3 can tell a field that was missing from a field that did not exist yet. Checked in the warehouse, a schema tells you six weeks later that a component has been emitting unanswerable signals. Checked in emit, it tells the engineer who wrote the component, in their own test run, before it merges.
why() and repeated_components are there for bugs rather than disputes. why() returns a list in emit order instead of a dictionary keyed by component. The book's controls_in uses a dictionary, and when I fed it six signals from one component in one run it returned one. The rule is one decision per component per proposal, so every control signal carries a proposal id in detail. A queue-drain run works through many tickets unattended and calls the refund gate once per ticket, as Part 1's drain_gated does. That is fine. A component that decides twice for one proposal is either a retry you know about or a graph node reached by two edges, and repeated_components is the query that finds it.
versions_on reads the counts, so a lightly used pod cannot be sampled out of its answer. The expected answer for a day is one harness version, or two on a deploy day. Two when nobody deployed usually means a pod that did not roll or a canary you forgot, though a config reload or a run resumed from a checkpoint can do it too. Either way, two sets of controls were in force at once.
Querying the agent decision log: why did this run not refund?
This code reruns the 02:14 case with the Second Stream present, on the repaired harness. In Part 3's design-doc stack the same 940.00 went through at 02:14, because no gate read the comparator. Here one does, and this repaired harness is the one legal asked about. Our intake sensor records what Part 4 taught it to record: the number of code points a step removed, never the removed text. The context assembler from Part 5 emits one OBSERVED signal per assembly. Its detail holds the evicted labels, the binding limit and assembled_tokens. A FloorBreached refusal becomes a control signal from a bound, and its reason is the breach kind: absent, over-cap or does-not-fit. Our comparator records the amount and digests of the account and the invoice rows it checked against. And the gate records the verdict it read.
KEY = b"held-by-the-system-of-record" # placeholder; a real key lives in a KMSAT = datetime(2025, 9, 10, 2, 14, 7, tzinfo=timezone.utc)TICKET = "I was charged twice for the September invoice. Please refund the duplicate."INVOICE_ROWS = ("inv_9107 team plan 960.00\ninv_9107 sso add-on 120.00\n" "inv_9107 sso add-on 120.00")def sig(run_id: str, kind: Kind, harness: str = "harness 0.6.0", **kw) -> Signal: return Signal(run_id=run_id, at=AT, kind=kind, harness_version=harness, model_id="acme/reasoner-3.1", **kw)def september_run(log: SignalLog) -> None: r = "r_01JB8" log.emit(sig(r, Kind.OBSERVED, component="intake", role=Role.SENSOR, outcome=Outcome.OBSERVED, reason="stripped our own quoted reply", detail={"ticket_body": digest(TICKET, KEY), "removed_code_points": "204"})) log.emit(sig(r, Kind.OBSERVED, component="context-assembler", role=Role.SENSOR, outcome=Outcome.OBSERVED, reason="evicted 1 span at its ceiling", detail={"evicted": "similar_tickets", "binding_limit": "retrieval-ceiling", "assembled_tokens": "24492"})) log.emit(sig(r, Kind.PROPOSED, detail={"proposal": "p1", "tool": "issue_refund", "amount": "940.00"})) log.emit(sig(r, Kind.CONTROL, component="daily-refund-bound", role=Role.BOUND, outcome=Outcome.WITHIN, reason="1,060.00 of 2,000.00 refunded today", detail={"proposal": "p1", "used": "1060.00", "ceiling": "2000.00"})) log.emit(sig(r, Kind.CONTROL, component="amount-on-invoice", role=Role.COMPARATOR, outcome=Outcome.FAILED, reason="940.00 matches no invoice row", detail={"proposal": "p1", "amount": "940.00", "account": digest("acct_4417", KEY), "invoice_rows": digest(INVOICE_ROWS, KEY)})) log.emit(sig(r, Kind.CONTROL, component="refund-gate", role=Role.GATE, outcome=Outcome.REFUSED, reason="escalated: amount-on-invoice failed", detail={"proposal": "p1", "amount": "940.00", "amount-on-invoice": "failed"})) log.emit(sig(r, Kind.ACTED, detail={"proposal": "p1", "tool": "escalate_to_human"})) # the run's read-only lookups would add their own ACTED signals; omitted herelog = SignalLog(SignalSchema(), boring_sample_rate=0.01, rng=random.Random(7))september_run(log)print("why did r_01JB8 not refund?")for s in log.why("r_01JB8"): print(f" {s.component:<20} {s.role.value:<11} {s.outcome.value:<9} {s.reason}")bound = log.why("r_01JB8")[0]print(f"one control signal as JSON: {len(json.dumps(asdict(bound), default=str))} bytes")log.emit(replace(bound, reason="1,060.00 of 2,000.00 refunded today (simulated retry)"))print(f"decided twice for one proposal: {log.repeated_components('r_01JB8')}")print(f"harness versions that decided that day: {log.versions_on('2025-09-10')}")why did r_01JB8 not refund? daily-refund-bound bound within 1,060.00 of 2,000.00 refunded today amount-on-invoice comparator failed 940.00 matches no invoice row refund-gate gate refused escalated: amount-on-invoice failedone control signal as JSON: 358 bytesdecided twice for one proposal: ['daily-refund-bound']harness versions that decided that day: {'harness 0.6.0'}Three lines answer legal's question from a query, and the answer contains the number the dispute turns on. The bound's line looks like noise. It is the line that proves the daily ceiling was checked on this run rather than skipped. This log samples boring records at 1%, and the line survived because the run proposed a refund. The careless log in the next section shows what happens without that rule. The last two lines are the bug queries run for real. A simulated retry of the bound shows up as a repeat for proposal p1, and the day's counts show one harness version in force.
Notice what the comparator's reason does not say. In the book's version of this example, the account id goes into the reason. An account id can be personal data, since GDPR Article 4(1) lists "an identification number" among the identifiers of a natural person. So this version puts it in detail as a digest. Even so, the reason field is the least policed field in the schema: f"cannot refund {ticket_body[:60]}" passes every check above. Treat reason as a reviewed field. Write it once per component, and read it in the pull request that adds it with one review question: does this f-string interpolate anything from a payload?
Five attempts follow, each a signal that would have been unanswerable or would have leaked. The schema refuses every one:
log = SignalLog(SignalSchema())base = dict(run_id="r_01JB8", at=AT, kind=Kind.CONTROL, harness_version="harness 0.6.0", model_id="acme/reasoner-3.1", component="refund-gate", role=Role.GATE)refused = dict(outcome=Outcome.REFUSED, reason="escalated")attempts = { "a refusal with no reason": dict(outcome=Outcome.REFUSED), "a reason that is only a label": dict(refused, reason="gate refused"), "an outcome nobody declared": dict(refused, outcome="refuse"), "the ticket text under its own name": dict(refused, detail={"ticket_body": TICKET}), "the ticket text under a new name": dict(refused, detail={"note": TICKET}),}for label, kw in attempts.items(): try: log.emit(Signal(**base, **kw)) print(f"{label}\n accepted") except SchemaViolation as exc: print(f"{label}\n SchemaViolation: {exc}")a refusal with no reason SchemaViolation: control signal from refund-gate: reason is requireda reason that is only a label SchemaViolation: control signal from refund-gate: reason is requiredan outcome nobody declared SchemaViolation: control signal from refund-gate: unknown outcome 'refuse'the ticket text under its own name SchemaViolation: ticket_body may only be recorded as a Digestthe ticket text under a new name SchemaViolation: note: 75 chars, limit 64The last line is the honest limit of the length cap. It caught a 75-character ticket. A 60-character one would pass, which is why the reviewed-field rule above still applies to detail keys a component invents.
The owed-decision check: catching a refund no control decided
Part 3 checked loop closure statically, in CI: every verdict is read by a gate, and every gate's inputs are emitted by something. That check cannot see a second dispatch path the graph does not declare, and Part 3 said so. The Second Stream can. For each effect, declare the controls that must decide before anything with that effect runs, and map every tool to an effect. An ACTED signal is then a finding unless every owed control recorded an allowing outcome for the same proposal, earlier in the same run. A missing decision is one kind of finding. The other is a refusal followed by the action anyway, which is the unread verdict Part 3 hunted statically.
One precondition decides whether this works. The ACTED signal must be written below the dispatch layer: by the tool wrapper at the credential, or better, reconciled from the payment processor's own disbursement feed. If the executor writes ACTED, a path that skips the executor skips the record too, and the check goes as blind as Part 3's wiring graph.
Both maps, EFFECTS and OWED, sit with SignalLog above, because the sampling rule derives CONSEQUENTIAL from them, and this check reads them too.
ROLE = {"daily-refund-bound": Role.BOUND, "amount-on-invoice": Role.COMPARATOR, "refund-gate": Role.GATE, "human-approval": Role.GATE}PASS = (("daily-refund-bound", Outcome.WITHIN), ("amount-on-invoice", Outcome.PASSED), ("refund-gate", Outcome.ALLOWED))REFUSE = PASS[:2] + (("refund-gate", Outcome.REFUSED),)def unbacked_actions(log: SignalLog) -> list[tuple[str, str]]: decided: dict[tuple[str, str], list[tuple[str, Outcome]]] = {} found = [] for s in sorted(log.records, key=lambda s: s.at): # stable: ties keep emit order key = (s.run_id, s.detail.get("proposal", "")) if s.kind is Kind.CONTROL: decided.setdefault(key, []).append((s.component, s.outcome)) continue tool = s.detail.get("tool", "") if s.kind is not Kind.ACTED: continue if tool not in EFFECTS: found.append((f"{s.run_id}/{key[1]} {tool}", "has no declared effect")) continue got = decided.get(key, []) owed = OWED[EFFECTS[tool]] missing = [c for c in owed if c not in {c for c, _ in got}] approved = ("human-approval", Outcome.ALLOWED) in got # may clear a gate only blocked = sorted({f"{c} {o.value}" for c, o in got if c in owed and o not in BORING and not (approved and ROLE.get(c) is Role.GATE)}) problems = ([f"no decision from {', '.join(missing)}"] if missing else []) + \ ([f"ran although {', '.join(blocked)}"] if blocked else []) lost = log.dropped[key] if missing and not blocked and lost >= len(missing): problems = [f"inconclusive: {lost} record lost in an outage"] if problems: found.append((f"{s.run_id}/{key[1]} {tool}", "; ".join(problems))) return founddef refund(log: SignalLog, run_id: str, proposal: str = "p1", tool: str = "issue_refund", decisions: tuple = PASS, lose: str = "") -> None: action = {"proposal": proposal, "tool": tool, "amount": "120.00"} log.emit(sig(run_id, Kind.PROPOSED, detail=action)) for name, outcome in decisions: signal = sig(run_id, Kind.CONTROL, component=name, role=ROLE[name], outcome=outcome, reason=f"{name}: 120.00 checked against inv_9107", detail={"proposal": proposal, "amount": "120.00"}) if name == lose: log.lose(signal) else: log.emit(signal) log.emit(sig(run_id, Kind.ACTED, detail=action)) # written by the tool wrapperdef drill(log: SignalLog) -> None: refund(log, "r_a") # every owed decision allowed it refund(log, "r_b", decisions=()) # the fast path refund(log, "r_c") # one run, two refunds, refund(log, "r_c", "p2", decisions=()) # only the first gated refund(log, "r_d", decisions=REFUSE) # refused, refunded anyway refund(log, "r_e", decisions=REFUSE + PASS[2:]) # retried until allowed refund(log, "r_f", tool="apply_credit", decisions=()) # same effect, other tool refund(log, "r_g", tool="goodwill_voucher", decisions=()) # effect never declared refund(log, "r_h", lose="refund-gate") # the store dropped a pass refund(log, "r_i", decisions=REFUSE + (("human-approval", Outcome.ALLOWED),)) refund(log, "r_j", tool="send_reply", decisions=()) # a promise, never checked refund(log, "r_k", decisions=(("daily-refund-bound", Outcome.WITHIN), ("amount-on-invoice", Outcome.FAILED), ("refund-gate", Outcome.REFUSED), ("human-approval", Outcome.ALLOWED)))log = SignalLog(SignalSchema(), boring_sample_rate=0.01, rng=random.Random(7))drill(log)for where, problem in unbacked_actions(log): print(f"{where}\n {problem}")careless = SignalLog(SignalSchema(), boring_sample_rate=0.01, rng=random.Random(7), spare_consequential=False)drill(careless)false_alarms = [w for w, _ in unbacked_actions(careless) if w.startswith("r_a/")]print(f"without sparing consequential runs, r_a raises {len(false_alarms)} false finding")sept = SignalLog(SignalSchema(), boring_sample_rate=0.01, rng=random.Random(7))september_run(sept)lost = sum(n for (run, _), n in sept.dropped.items() if run == "r_01JB8")findings = unbacked_actions(sept)print(f"september run r_01JB8: {len(findings)} findings, {lost} records lost")sent = [s for s in sept.records if s.kind is Kind.ACTED and s.detail.get("tool") == "send_reply"]print(f"send_reply actions in r_01JB8: {len(sent)}")r_b/p1 issue_refund no decision from daily-refund-bound, amount-on-invoice, refund-gater_c/p2 issue_refund no decision from daily-refund-bound, amount-on-invoice, refund-gater_d/p1 issue_refund ran although refund-gate refusedr_e/p1 issue_refund ran although refund-gate refusedr_f/p1 apply_credit no decision from daily-refund-bound, amount-on-invoice, refund-gater_g/p1 goodwill_voucher has no declared effectr_h/p1 issue_refund inconclusive: 1 record lost in an outager_j/p1 send_reply no decision from reply-claims-checkr_k/p1 issue_refund ran although amount-on-invoice failedwithout sparing consequential runs, r_a raises 1 false findingseptember run r_01JB8: 0 findings, 0 records lostsend_reply actions in r_01JB8: 0r_a and r_b are the runs from wrong_way.py, now with the Second Stream. Both refunded 120.00, and the log no longer treats them as the same. r_c is the case a per-run check would miss: one run, two refunds, and the gated first one does not vouch for the second. r_d is a refusal the code ignored. r_e is the quieter version, a refusal retried until the gate said yes, which a check that kept only the last outcome would pass.
r_f and r_g are the agent-specific cases. The model reached a money-out effect through apply_credit, which a registration keyed by issue_refund would never examine. Then it called a tool nobody had mapped to an effect at all. r_h is the completeness case. Its gate passed, but the store lost that record, so the finding is inconclusive rather than a bypass. r_i raises nothing. Its comparator passed, the gate's own rule said no, the run escalated, and a person approved the refund. The check hard-codes one override: a human approval clears a gate's refusal, and only a gate's. It cannot clear a bound's exceeded, which is why a bound, which never trusts the proposal, has no override. It cannot clear a comparator's factual verdict either. r_k is the September run with a person's approval added, and it still pages, because 940.00 is on no invoice. To refund that amount, correct the invoice first. r_j is the promise effect: a send_reply with no decision from reply-claims-check.
The next line shows the same coupling for sampling. Sample the traffic without sparing consequential runs, and the honest run r_a looks like a bypass. The last line runs the check on the September night itself. In the diagram, the left column is what a backed action looks like and the right column is what the query pages on.
grid-rows: 1
grid-gap: 40
ra: "r_a: gate called" {
label.near: top-center
grid-columns: 1
grid-gap: 12
style: {fill: "#EFF9F1"; stroke: "#3F9E52"; font-color: "#2C2C2A"}
a1: "PROPOSED issue_refund 120.00" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
a2: "CONTROL x3: within, passed, allowed" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
a3: "ACTED issue_refund" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
a4: "backed: every owed decision\nallowed it" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
}
rb: "r_b: fast path" {
label.near: top-center
grid-columns: 1
grid-gap: 12
style: {fill: "#FDEDEC"; stroke: "#B03A2E"; font-color: "#2C2C2A"}
b1: "PROPOSED issue_refund 120.00" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
b2: "no CONTROL signals" {style: {fill: "#95A5A6"; stroke: "#6F7C7D"; font-color: "#2C2C2A"}}
b3: "ACTED issue_refund" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
b4: "finding: 3 owed decisions\nmissing" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
}
Both columns hold the same proposal and the same action, so the only difference is the three control signals in the middle. Your effect map has to come from the executor's own registration, not from a second list kept by hand. Part 3 gave the reason: two lists drift.
Run the check in two places. Inline, the tool wrapper at the credential can refuse to call the processor unless the owed decisions for this proposal are on record. That is complete mediation proper. "On record" should not mean a synchronous read of the signal store, or a store outage would fail every refund closed. Have the gate hand the wrapper a signed decision token, an HMAC over the proposal id, the arguments and the owed decisions, and have the wrapper verify the token. The scheduled query stays as the independent check for anything that reached the processor another way, so its action records should come from the processor's own feed. That feed carries a run id and proposal id only if you pass them through. Put them in the idempotency key or the metadata on the refund call. Without them, every processor-fed refund looks unbacked. Those records now come from a second writer, so order them by at. Give late records a grace window of a few minutes before a finding pages, and compare the processor's timestamps with a skew tolerance or use the harness's own receive time.
Two rules keep the check from paging on honest traffic or missing dishonest traffic. A refusal or a raised is final for its proposal, except a gate refusal cleared by a recorded human-approval: a legitimate retry gets a new proposal id, so r_e's retry-until-allowed is a finding by design. And the check matches decisions to a proposal id, not to what the action did. Put the amount, or a keyed digest of the arguments, on each control signal, and flag an ACTED whose arguments differ or a second ACTED for one proposal. Using the proposal id as the processor's idempotency key already blocks the double refund at the processor; Part 1 deferred the rest of idempotency to Part 12.
Updated: Part 9 of this series, on gated execution, narrows this override. The "gate refusal" a human approval may clear is an escalation inside the review band of the mode the gate decided in, whose required verdicts passed and which offered "approve". A refusal is final for its proposal, so the check should clear an approval only when the gate's signal records all three.
Neither an OPA decision log nor a trace backend can support this check on its own. The registration is what says which decisions were owed, and neither of them holds that list. The transport is a separate question. You can carry the Second Stream as OpenTelemetry log records, routed by a collector to their own exporter with no sampling and their own retention. What the harness must own is the schema, the check at emit, the counts, the owed decisions and the retention, not the pipe.
Sampling an agent decision log without losing absence
Question 4 of the Six-Month Question, "has this control ever done anything", is the one the volume argument attacks. A system doing 100,000 runs a day with eight controls emits 800,000 control signals a day. Nearly all of them say a bound was within its ceiling or a comparator passed. Usually the advice is to sample them away. OpenTelemetry's sampling guide puts it plainly: if most requests succeed, "you do not need 100% of your traces to meaningfully observe your applications and systems" (OpenTelemetry sampling). OPA's own documentation shows a drop rule that "will drop all requests to the allow rule in the kafka package, that returned true" (OPA decision logs).
For run facts that advice is right. For control facts it destroys the one property the boring signals exist for: being able to tell a control that has never fired from a control that is not there. It also breaks the owed-decision check. That check reads a missing pass as a bypass, as the careless log showed.
Do the arithmetic before arguing about it. One control signal serialises to 358 bytes of JSON, as the query above printed. Eight hundred thousand of those is about 290 MB a day. Kept for 400 days, that is about 115 GB. Sensor, proposal and action signals raise the count to roughly 13 to 15 per run instead of 8. Assuming they are about the same size, an estimate, that puts the stream closer to 500 MB a day. That is still small next to full-content traces of the same traffic. Retention is what becomes a real warehouse line item, and retention is what answers the March question.
So the rule has two parts. Never sample a run that proposed a consequential tool. The per-run answer to "why was this refund allowed" is exactly the question a fraud review asks. Sample the boring records of everything else if you must, and keep every count. SignalLog does both.
Many control decisions in the billing agent guard reads, not writes. account-scope-gate is one of them: it refuses an invoice lookup for another customer's account. The code below runs it over 10,000 runs with three refusals. It samples boring outcomes at 1%, and it applies 1% trace-level sampling to the same runs for comparison:
def traffic(log: SignalLog, runs: int = 10_000) -> None: for i in range(runs): refused = i in (1_811, 4_072, 9_530) log.emit(Signal(run_id=f"r_{i:05d}", at=AT + timedelta(minutes=i), kind=Kind.CONTROL, harness_version="harness 0.6.0", model_id="acme/reasoner-3.1", component="account-scope-gate", role=Role.GATE, outcome=Outcome.REFUSED if refused else Outcome.ALLOWED, reason="invoice belongs to another account" if refused else "invoice belongs to this account"))kept = SignalLog(SignalSchema(), boring_sample_rate=0.01, rng=random.Random(7))traffic(kept)print(f"records kept: {len(kept.records)} of 10000")print(f"refusals on record: {sum(s.outcome is Outcome.REFUSED for s in kept.records)}")gate = "account-scope-gate"counted = sum(kept.decided(gate, o) for o in (Outcome.ALLOWED, Outcome.REFUSED))print(f"decisions counted: {counted}")trace_level = random.Random(7)survivors = [i for i in range(10_000) if trace_level.random() < 0.01]print(f"1% trace sampling keeps {len(survivors)} runs, refusals among them: " f"{sum(i in (1_811, 4_072, 9_530) for i in survivors)}")print(f"chance 1% trace sampling keeps any of 3 refusals: {1 - 0.99 ** 3:.1%}")records kept: 109 of 10000refusals on record: 3decisions counted: 100001% trace sampling keeps 106 runs, refusals among them: 0chance 1% trace sampling keeps any of 3 refusals: 3.0%grid-rows: 1
grid-gap: 40
by_outcome: "Sampled by outcome (the Second Stream)" {
label.near: top-center
grid-columns: 1
grid-gap: 14
style: {fill: "#EFF9F1"; stroke: "#3F9E52"; font-color: "#2C2C2A"}
l1: "10,000 read-gate decisions" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
l2: "counts: 10,000 (exact)" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
l3: "records: 109\nall 3 refusals kept" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
}
by_trace: "Head-sampled by trace (1%)" {
label.near: top-center
grid-columns: 1
grid-gap: 14
style: {fill: "#FDEDEC"; stroke: "#B03A2E"; font-color: "#2C2C2A"}
t1: "10,000 runs" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
t2: "counts: none in the SDK" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
t3: "records: 106 runs\n0 of 3 refusals kept" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
}
The record volume is almost the same on both sides, 109 against 106. Everything else differs. Sampling by outcome keeps every refusal and an exact count of everything, so a counter answers "has the scope gate decided anything this month" exactly.
Head sampling is the mechanism the Langfuse SDK and OpenTelemetry's SDK samplers use once you turn sampling on. It decides before it knows the outcome, so it drops a refusal exactly as readily as a pass. With three refusals in 10,000 runs, it keeps any of them 3.0% of the time. The zero above is the expected result, not an unlucky seed. Langfuse documents this behaviour directly: "If a trace is not sampled, none of its observations or associated scores will be sent" (Langfuse sampling).
A tail sampler in an OpenTelemetry Collector is different. It decides after the outcome, and a policy that matches an attribute such as the tool name can keep every refund trace (tail sampling processor). So capability is not the difference. Retention and payload are. A kept trace carries the prompts and tool results on trace retention, while this stream carries 358-byte decisions on a retention somebody chose. The advice to drop allows, in OpenTelemetry's guide and in OPA's drop rules, is still unsafe for consequential runs, because the owed-decision check reads a dropped allow as a bypass.
Here I part slightly from the book. It tells you never to drop the within-ceiling and passed outcomes, and its own production section then concedes that counts are enough to keep absence distinguishable from zero. I think the counts are the actual rule.
Why payload digests in an agent log must be keyed
The signal stream outlives the run, the incident, the ticket and often the customer relationship. It is copied into a warehouse and read by people with no connection to the original request, so it is no place to keep the ticket body. OpenTelemetry's GenAI conventions reach the same conclusion for content: instrumentations "SHOULD NOT capture them by default," and for production the recommended pattern is to "Store content externally and record references on the spans" (GenAI spans). So the real "trace everything" default comes from vendors and practitioners. The OpenAI Agents SDK documents that "By default, trace_include_sensitive_data is True" (tracing docs). Hightower's harness series gives the same advice for tools: "capture what the tool returned, not just that it was called" (Hightower, 2026). Kubernetes offers the other way in its audit policy. An operator can choose, per rule, a Metadata level that records events "with metadata (requesting user, timestamp, resource, verb, etc.) but not request or response body" (Kubernetes auditing).
A digest answers the questions that actually get asked, without the text. Did two runs see the same ticket? Did the assembled context change between turn eight and turn nine? Is the knowledge-base article the model read the one that was edited last week? All three are comparisons, and none needs the content. When a postmortem does need the text, the original lives in the system of record with its own retention and access controls. Its run id and ticket id locate it there. The digest proves the copy you fetched is the one the run saw. It can only confirm a copy you already found. That is the same split Part 4 made for the raw ticket.
The book's digest() is an unkeyed SHA-256 truncated to 64 bits, and that is the one choice in its schema I would change before production. An unkeyed hash of a short or structured field can be reversed by guessing. Account ids like acct_4417 are a few thousand to a few million candidates, and my laptop hashed a million of them with SHA-256 in 1.4 seconds. The United States Federal Trade Commission (FTC) made the same point in a post titled "No, hashing still doesn't make your data anonymous" (FTC Office of Technology). It also misses the General Data Protection Regulation (GDPR) definition of pseudonymisation, which requires "additional information" that is "kept separately" (GDPR Art. 4). An unkeyed hash has no additional information to keep. A keyed HMAC does: the key.
direction: down
key: "HMAC key, held by the system of record\ndelete it: every digest becomes unresolvable" {style: {fill: "#C2185B"; stroke: "#8E1243"; font-color: "#FFFFFF"}}
reader: "Investigator\nin March" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
stream: "Second Stream\ndigests + counts, 400 days" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
sor: "System of record\nticket text, invoice rows\nown retention + access" {style: {fill: "#955D37"; stroke: "#6E4426"; font-color: "#FFFFFF"}}
key -> stream: "1. digest at emit"
reader -> stream: "2. query decisions"
stream -> sor: "3. fetch by ticket id,\ncheck the digest (key + access)"
The diagram shows what the key buys beyond resistance to guessing. The stream and the store keep their own lifetimes. The key lives in a key management service, where the harness can compute digests with it but cannot export it, and warehouse readers have no access to it at all. So testing a guess against a digest needs the key, and the people reading 400 days of rows do not have it. Rotation has a cost: digests made before and after a rotation no longer match, so "did two runs see the same ticket" only works within one key's lifetime. Delete the key, or rotate it out after the store's retention period, and nobody can test a guess against those digests again. They still link runs to each other, so whether that alone satisfies GDPR storage limitation is a question for your data protection officer. It is still much cheaper than rewriting 400 days of warehouse rows. It is also consistent with Part 4, which already put a keyed HMAC of the raw bytes in its record.
When the schema check fails: arguing with OpenTelemetry's error principle
Raising SchemaViolation inside emit means a telemetry problem can fail a request. That runs straight against OpenTelemetry's design principle: "We assume that users would prefer to lose telemetry data rather than have the library significantly change the behavior of the instrumented application," and implementations "MUST NOT throw unhandled exceptions at runtime" (OpenTelemetry error handling). For run facts that principle is correct, and I would not argue with it.
For the controls' own record I think it is the wrong default, for one reason. A malformed control signal means the control layer has quietly stopped being answerable, in the one place you would look to see whether the controls work. The diagram splits the failure into three cases, because a schema violation and a store outage get different answers.
grid-rows: 3
grid-columns: 2
grid-gap: 22
q1: "1. Signal fails the schema" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
a1: "raise in emit: a code defect,\nfound in the author's test run" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
q2: "2. Store down, copilot mode" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
a2: "drop the record, count the drop,\nalert on the count" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
q3: "3. Store down, queue-drain mode" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
a3: "fail the run: nobody is there\nto remember what happened" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
q1 -> q2
q2 -> q3
A schema violation is a bug in the component that emitted it, so it should be as loud as any other bug. If the component has a test, the violation fires in that test, where it costs nothing.
An unavailable store is an operational event, and the right response depends on who is waiting. The billing agent runs in two modes. In copilot mode a human agent reviews each reply before it goes out. A person is waiting, and an imperfectly recorded reply beats none, so drop the record and count the drop somewhere that survives the outage. The owed-decision check reads that count, which is why r_h came back inconclusive instead of paging as a bypass. In queue-drain mode the agent works through the ticket queue unattended. Nobody is watching, and an unrecorded decision there is the most expensive kind.
What you must not do is what happens by default. Somebody adds a try/except so that telemetry can never take down production, and it turns a missing record into silence.
One caveat on "a bug found in a test": the size caps depend on data, so a long eviction list can breach them in production with no test ever seeing it. Raise on structural violations, a missing field, an unknown outcome or raw text under a never-raw key. Treat a size breach like an outage, with a counted, truncated record.
My defaults are fixed per mode: raise on a structural schema violation everywhere, drop and count in copilot mode, fail the run in queue-drain mode. A measured cost would change the last one. If runs failed by store outages cost more than the disputes their missing records would have answered, move queue-drain to drop and count too. Never accept a malformed signal, because a malformed record in the store looks like evidence.
Harness replay: re-running today's controls on recorded decisions
You cannot reproduce September's model call. The model has been updated, it ran at a temperature, and the prompt has been edited twice since. A re-run produces a different output, and the comparison tells you nothing you can act on. Controls are different. They are deterministic code you own, the stream records what each decided and on which inputs, and today's version of each can be asked the same question.
Durable-execution engines already split work this way. Temporal's documentation puts "LLM/AI invocations" in activities that "execute outside the replay path," and replays only the deterministic workflow code (Temporal workflow definition). OPA records non-deterministic built-in results in nd_builtin_cache, "Intended for use in debugging and decision replay" (OPA decision logs). Harness replay differs from Temporal's in one way that matters. Temporal replays unchanged code and treats a mismatch as an error. Harness replay runs changed controls on recorded inputs and treats a mismatch as information.
A control can be replayed only as far as its inputs are in detail, or one pointer away from it. That is why the refund gate above records the comparator verdict it read. The book's demo gate records only the tool and the amount, so it could not actually be replayed. The report has four outcomes, and an empty corpus is reported as empty rather than as "no divergence":
@dataclassclass ReplayReport: examined: int = 0 unchanged: int = 0 diverged: list[tuple[str, str, str]] = field(default_factory=list) not_replayable: list[tuple[str, str]] = field(default_factory=list) absent: list[str] = field(default_factory=list)Control = Callable[[Mapping[str, str | Digest]], "Outcome | str"]def replay(recorded: Iterable[Signal], today: Mapping[str, Control]) -> ReplayReport: report = ReplayReport() for s in recorded: if s.kind is not Kind.CONTROL: continue report.examined += 1 control = today.get(s.component) if control is None: report.absent.append(s.component) continue now = control(s.detail) if not isinstance(now, Outcome): report.not_replayable.append((s.component, now)) elif now is s.outcome: report.unchanged += 1 else: report.diverged.append((s.component, s.outcome.value, now.value)) return reportdef daily_bound_today(detail): if Decimal(detail["used"]) < Decimal(detail["ceiling"]): return Outcome.WITHIN return Outcome.EXCEEDEDdef amount_on_invoice_today(detail): rows = detail.get("invoice_rows") if isinstance(rows, Digest): return "needs invoice_rows from the system of record" amounts = {Decimal(line.split()[-1]) for line in rows.splitlines()} return Outcome.PASSED if Decimal(detail["amount"]) in amounts else Outcome.FAILEDdef refund_gate_today(detail): if detail.get("amount-on-invoice") == "passed": return Outcome.ALLOWED if Decimal(detail["amount"]) < 1000: # the goodwill rule added in January return Outcome.ALLOWED return Outcome.REFUSEDTODAY = {"daily-refund-bound": daily_bound_today, "amount-on-invoice": amount_on_invoice_today, "refund-gate": refund_gate_today}def show(report: ReplayReport) -> None: if report.examined == 0: print("examined 0 control decisions: nothing to compare, not 'unchanged'") return print(f"examined {report.examined} control decisions") print(f" unchanged {report.unchanged}") for c, then, now in report.diverged: print(f" diverged {c:<20} then={then:<8} now={now}") for c, why in report.not_replayable: print(f" not replayable {c:<20} {why}") for c in report.absent: print(f" absent {c}")log = SignalLog(SignalSchema())september_run(log)show(replay(log.records, TODAY))show(replay([], TODAY))show(replay(log.records, {k: v for k, v in TODAY.items() if k != "amount-on-invoice"}))examined 3 control decisions unchanged 1 diverged refund-gate then=refused now=allowed not replayable amount-on-invoice needs invoice_rows from the system of recordexamined 0 control decisions: nothing to compare, not 'unchanged'examined 3 control decisions unchanged 1 diverged refund-gate then=refused now=allowed absent amount-on-invoicedirection: down
rec: "Recorded control signals\n(September, harness 0.6.0)" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
now: "Today's controls\n(the model is left out)" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
out: "one outcome per recorded decision" {
grid-rows: 1
grid-gap: 18
label.near: bottom-center
style: {fill: "#F4F6F6"; stroke: "#6F7C7D"; font-color: "#2C2C2A"}
u: "unchanged: 1\ndaily-refund-bound" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
d: "diverged: refund-gate\nrefused then, allowed now" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
n: "not replayable:\namount-on-invoice\n(input is a digest)" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
a: "absent: amount-on-invoice\n(after deleting it)" {style: {fill: "#95A5A6"; stroke: "#6F7C7D"; font-color: "#2C2C2A"}}
}
rec -> now: "detail only"
now -> out
Each outcome means something different in a dispute. Unchanged says today's harness would do what September's did, which is the answer that closes most disputes. Diverged says somebody changed the rule: the January goodwill exception would now refund 940.00 without a matching invoice row. That may be the right policy. It is now a fact with a date, which is what you want in front of you when a customer argues a decision made under the old rules. Not replayable says the decision turned on an input the stream holds only as a digest, so replaying it needs one authorised fetch from the system of record. Absent, in the third call after deleting the comparator, says the control that made the decision no longer exists. That is how you learn a refusal in September came from a component nobody can find in March.
Of all these fixes, the empty-corpus line is the one I care most about. The book's replay() returns a list of divergences. An empty corpus, a sampled-away week or a run that emitted no control signals all return [], which reads as "today's harness agrees". That is the absence-means-nothing failure again, inside the tool meant to detect it.
One limit applies to every row. Each control is replayed on what was recorded, so refund_gate_today reads September's comparator verdict, not today's. If the comparator changed too, the answer for the whole chain needs the comparator replayed first and its new verdict fed to the gate, in executor order.
Replay is also useful outside disputes. Before a gate change, replay a week of recorded decisions against the new code and count the divergences. A change that flips 11% of last week's refusals has a known blast radius before it ships. It is also the cheaper relative of an adversarial test suite: real traffic, real decisions, and a check that today's harness still makes them. And it is different from replaying recorded prompts through a new model, which tests the half of the system the harness does not own.
Answering legal's question: what the queries return
Go back to legal's question about 10 September. With the Second Stream, the answer is a query that returns three lines carrying the amount, the harness version and the model id. A replay against today's controls shows the January goodwill rule would now refund the same 940.00. Run on that night's records above, the owed-decision check finds that no money-out or promise effect ran in that run, so nothing was owed and nothing was missing, and no record was lost. That also bounds the customer's allegation, as far as the stream can. With ACTED written at the credential, a send_reply that promised a refund would appear in this run with its own owed decision, as r_j does in the drill. None appears, so the approval message did not come from this run's tool calls. Next, look at the reply written after the human escalation. Unless you emit control spans into it, a trace gives you none of those for the controls a harness owns, and a head-sampled trace cannot give you the third. A decision log alone gives you the first two. The third needs owed decisions declared per effect and a log whose completeness the sampling and outage policies protect. That third answer is the part this article adds: owed decisions declared per effect, so an undeclared tool is a finding at run time, and sampling and outage policy set by the log's completeness. It is the reason to own this log in the harness rather than wait for the tracing stack to grow one.
What would change this: when a separate agent decision log stops being necessary
- OpenTelemetry ships a GenAI convention for control decisions with an outcome, a reason and a version per decision, and vendors sample it by outcome. Then the Second Stream becomes a set of attributes on a span, and you should adopt the convention rather than maintain your own schema. As of November 2026, the repository has
gen_ai.evaluation.resultand no release. - Framework guardrail spans start carrying a reason and a version, and frameworks instrument the business checks your code makes. The fields argument would then be settled by the framework, and only retention and sampling would remain as reasons to keep a separate stream.
- Model self-explanation becomes reliable evidence. Some teams already prompt the agent to explain its own decisions (hazbeen). After the Replit agent deleted a production database during a code freeze, Fortune reported that its user came to believe "the AI had potentially fabricated its response" about whether a rollback would work (Fortune, 2025). Until a model's account of itself is evidence, the controls' record is the only first-hand witness to why.
Checklist: what your harness must emit
- Every comparator, gate and bound emits one control signal per proposal it decides, with the proposal id in
detail, on the happy path too. Find the ones that emit only on failure. - Every signal carries
harness_versionandmodel_id, and the log stamps the schema version. Nothing is inferred from a run-started event. -
outcomeis a closed enum checked at emit, andreasonmust be more than the role and outcome label. - Payload fields can only be written as a keyed digest. The key lives with the system of record, and plain
detailvalues have a length cap. -
reasonstrings are reviewed in the pull request that adds the component, with one question: does this interpolate a payload? - Runs keep every record from the moment they propose a consequential tool. Boring outcomes elsewhere may be sampled, but every outcome is counted per component, harness version and day before sampling.
- The executor's registration declares owed decisions per effect (money out, a promise to a customer) and maps every tool to an effect. A tool with no declared effect is a finding.
- A scheduled query finds every
ACTEDsignal without an allowing decision for each of its owed decisions, for the same proposal, including a refusal that was retried until it passed, and pages on any result. Runs with dropped records come back inconclusive, not as bypasses. - The tool wrapper at the credential also refuses inline unless the owed decisions are on record. The run id and proposal id travel to the processor, as the idempotency key or metadata, so its feed can be reconciled.
-
ACTEDis written by the tool wrapper at the credential, or reconciled from the processor's disbursement feed, never by the executor alone. Records are ordered byat, with a grace window for late ones. - Day-one queries exist: why did run X do that, has this control refused anything in 30 days, how many harness versions decided anything on a given day, which components decided twice for one proposal, which emitted
raised. - Schema violations raise in
emit. A store outage has a written policy per mode (drop and count, or fail the run), and no silenttry/except. - Each control's
detailholds enough to re-decide, or a digest one fetch away. Replay reports0 examinedseparately from0 diverged. - Retention is a number somebody chose, checked against the longest Six-Month Question you could be asked. 400 days covers a September run asked about in the following October. If the audit trail is a filtered view over this stream with a longer retention, as in the AI control plane audit trail post, set that number separately. By analogy, some regulated reasons need longer: Regulation B keeps the reasons for an adverse credit decision for 25 months, or 12 months for business credit (12 CFR 1002.12).
Going deeper: Chapter 6 of the book
This is Part 6 of a series that follows Harness Engineering for Production AI Systems one chapter at a time. Chapter 6, "Signals, and What the Harness Must Emit," builds the signal schema as a TOML policy file. It wires a recorder into the executor from Part 3, so the readiness check later in the book reads production rather than a test fixture. It also covers run identity across checkpoints and sub-agents, the audit trail as a filtered view over the stream rather than a second system, and why the platform team should own the pipe while the harness team owns the schema. Code for the chapter is the ch06-signals tag of the companion repository. Its tests need no model and no API key. The never-raw prefix check, the per-value detail cap, the unstamped schema version, the collapsing controls_in, the silent empty replay and the unkeyed digest are logged for the book's next edition.
References
- Open Policy Agent. Decision Logs. Accessed 2026-11-13. https://www.openpolicyagent.org/docs/management-decision-logs
- Kubernetes. Audit Annotations. Accessed 2026-11-13. https://kubernetes.io/docs/reference/labels-annotations-taints/audit-annotations/
- Kubernetes. Auditing. Accessed 2026-11-13. https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/
- OpenTelemetry. Semantic conventions for generative AI client spans (semantic-conventions-genai, Development status, main branch). Accessed 2026-11-13. https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-spans.md
- OpenTelemetry. GenAI events registry (
model/gen-ai/events.yaml). Accessed 2026-11-13. https://github.com/open-telemetry/semantic-conventions-genai/blob/main/model/gen-ai/events.yaml - OpenTelemetry. Sampling. Accessed 2026-11-13. https://opentelemetry.io/docs/concepts/sampling/
- OpenTelemetry. Error handling in OpenTelemetry (specification). Accessed 2026-11-13. https://opentelemetry.io/docs/specs/otel/error-handling/
- OpenAI. Agents SDK tracing (docs source, main). Accessed 2026-11-13. https://github.com/openai/openai-agents-python/blob/main/docs/tracing.md
- OpenAI.
openai-agents-pythonsrc/agents/tracing/span_data.py(GuardrailSpanData), main. Accessed 2026-11-13. https://github.com/openai/openai-agents-python/blob/main/src/agents/tracing/span_data.py - OpenAI.
openai-agents-pythonsrc/agents/run_internal/guardrails.py, main. Accessed 2026-11-13. https://github.com/openai/openai-agents-python/blob/main/src/agents/run_internal/guardrails.py - Langfuse. Observation Types. Accessed 2026-11-13. https://langfuse.com/docs/observability/features/observation-types
- Langfuse. Sampling. Accessed 2026-11-13. https://langfuse.com/docs/observability/features/sampling
- Langfuse. Releases & Versioning. Accessed 2026-11-13. https://langfuse.com/docs/observability/features/releases-and-versioning
- Arize. OpenInference Semantic Conventions. Accessed 2026-11-13. https://github.com/Arize-ai/openinference/blob/main/spec/semantic_conventions.md
- Amazon Web Services. ApplyGuardrail, Amazon Bedrock API Reference. Accessed 2026-11-13. https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_ApplyGuardrail.html
- OpenTelemetry. Tail Sampling Processor (opentelemetry-collector-contrib). Accessed 2026-11-13. https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor
- Temporal. Workflow Definition. Accessed 2026-11-13. https://docs.temporal.io/workflow-definition
- Saltzer, J. H., & Schroeder, M. D. (1975). The Protection of Information in Computer Systems. Proceedings of the IEEE, 63(9). https://www.cs.virginia.edu/~evans/cs551/saltzer/
- European Union. Regulation (EU) 2016/679 (GDPR), Article 4 (definitions 4(1) personal data and 4(5) pseudonymisation). Accessed 2026-11-13. https://gdpr-info.eu/art-4-gdpr/
- US Federal Trade Commission, Office of Technology (2024). No, hashing still doesn't make your data anonymous (title from the page URL; the page blocked automated fetches). https://www.ftc.gov/policy/advocacy-research/tech-at-ftc/2024/07/no-hashing-still-doesnt-make-your-data-anonymous
- US Consumer Financial Protection Bureau. 12 CFR 1002.12 Record retention (Regulation B). Accessed 2026-11-13. https://www.consumerfinance.gov/rules-policy/regulations/1002/12/
- Fortune (2025-07-23). AI-powered coding tool wiped out a software company's database in 'catastrophic failure'. Accessed 2026-11-13. https://fortune.com/2025/07/23/ai-coding-tool-replit-wiped-database-called-it-a-catastrophic-failure/
- Hightower, R. (2026). Harness Engineering - Observability: The Green Dashboard and the Seven Customers Who Were Never Booked. Accessed 2026-11-13. https://rickhigh.substack.com/p/harness-engineering-observability
- Böckeler, B. (2026). Harness engineering for coding agent users. martinfowler.com. Accessed 2026-11-13. https://martinfowler.com/articles/harness-engineering.html
- hazbeen (2026). Experiments on agent decision observability. Accessed 2026-11-13. https://hazbeen.substack.com/p/experiments-on-agent-decision-observability
- Kumar, R. (2026). AI control plane observability for agent fleets and AI control plane audit trails and regulatory requirements. ranjankumar.in. https://ranjankumar.in/ai-control-plane-unified-observability-agent-fleet , https://ranjankumar.in/ai-control-plane-compliance-audit-trails-regulatory-requirements
- Kumar, R. (2026). Agent Audit Trails: Logging Context, Not Just Actions. ranjankumar.in, 23 February 2026. https://ranjankumar.in/agent-audit-trails-logging-context-not-just-actions
- Kumar, R. (2026). Parts 1-5 of this series: Part 1, Part 2, Part 3, Part 4, Part 5. ranjankumar.in.
- Kumar, R. (2026). Harness Engineering for Production AI Systems. Chapter 6, "Signals, and What the Harness Must Emit." https://harness.ranjankumar.in/
- Kumar, R. (2026). harness-engineering-code, tag
ch06-signals. https://github.com/ranjankumar-gh/harness-engineering-code/tree/ch06-signals
Related Articles
- AI Agent Guardrail Monitoring: A Refusal Is Not Proof of Life
- LangGraph Guardrails: A Node That Skips the Executor
- LLM Output Repair: A Re-ask That Passes Can Still Be a Guess
- LLM Fallback Models Inherit Authority They Never Earned






