Guides · Series · 18 parts
Harness Engineering for Production AI Systems
The harness is a control system. Every component in it is a sensor, a comparator, a gate, or a bound — and anything else is application code.
The book
Harness Engineering for Production AI Systems
Sensors, Gates, and Bounds for LLM Applications That Have to Work When the Model Doesn't
Build the control layer around your LLM application: sensors that observe, comparators that check, gates that decide, and bounds that cap.
Four roles, and nothing else
Most writing about making LLM applications reliable is a list of techniques: validate the JSON, add a retry, put a human in the loop. Lists do not tell you what is missing. This series starts from a different claim — the harness around a model is a control system, and every component in it plays exactly one of four roles. A sensor observes. A comparator checks an observation against a standard. A gate decides whether a proposal becomes an action. A bound caps how far the system can go. If a component is none of these, it is not harness.
That claim is useful because it is checkable. Classify every component you have, and the gaps show up as roles nobody fills: a gate with no comparator behind it, a bound that only records spending after it happens, a sensor whose signal no one reads.
One agent, seventeen chapters
Every part follows the same system: a billing support agent for a mid-size B2B software company, running in two modes — copilot, with a human in the loop on every reply, and queue-drain, working the backlog overnight with nobody watching. It can read tickets and invoices, and it can post replies, apply credits, and issue refunds. Each part takes one chapter of the book, names the failure that chapter exists to prevent, and argues one position about it.
This is the book-length treatment. The earlier eight-part Harness Engineering series covers several of the same layers more briefly; this one goes chapter by chapter, adds the parts that series never reached — signals, cost bounds, authority bands, testing and attributing failures — and ends with the chapter on where the harness meets the framework.
Which part solves my problem
The series reads in order, but if something is already broken, start here.
- “The demo worked and production does not, and the fix everyone proposes is a better model.”Part 1 →
- “I cannot say which of my components actually decides anything.”Part 3 →
- “The agent’s own earlier reply came back inside a customer ticket and was treated as the customer’s words.”Part 4 →
- “Context compaction dropped a rule out of the system prompt thirty turns in.”Part 5 →
- “A model upgrade changed the output format and the validator now rejects everything — and nobody can say which layer failed.”Part 16 →
- “The provider returned 503 for four minutes and the agent retried into a storm.”Part 8 →
- “The model proposed a refund for an amount that appears on no invoice.”Part 9 →
- “A knowledge-base article carried an instruction addressed to the agent, and every gate passed it.”Part 11 →
- “The process restarted and I cannot tell which side effects already happened.”Part 12 →
- “Unattended mode quietly inherited the authority of the supervised mode.”Part 14 →
- “The circuit breaker stuck open and nobody noticed, because escalation looks like success.”Part 17 →
The series
0 of 18 published
The demo cliff is not a model problem
Why the gap between the inputs a demo was shown and the inputs production meets is a property of the input distribution, not the model — and the debt you take on with Borrowed Evidence measured in a domain that has an oracle.
Agent = model + harness, and the trust boundary
The Flat Context, where every token arrives on the same footing and provenance is something your code carries or nobody has, and the Plant Model: the model’s measured behaviour distribution, recorded as a dated document that bounds are designed against.
Sensors, comparators, gates, bounds
The Role Test that classifies any component as sensor, comparator, gate, bound, or application code; Closure, the static property that every verdict is read by some gate; and the Harness Residual that fails the test and is not harness at all.
Input defense and normalisation
Laundered Provenance, where text re-enters the system under a new origin with no record of the change — commonly your own output returning as input — and the Normalisation Window, the one interval in which the raw bytes still exist to be checked.
Context assembly as managed infrastructure
The Context Floor below which the harness refuses at assembly time rather than degrading, and Position Debt: the gap between a span being present in the window and a span being used.
Signals, and what the harness must emit
The Second Stream — the record of what each control decided and why, which no volume of run traces can reconstruct — and the Six-Month Question a signal schema is designed backwards from.
Validation and repair
The Repair Ladder, ordered by how much information and cost each rung spends, and the Deferral Rung: reporting that no valid result exists and stopping, because a ladder that always answers guesses when it should defer.
Retry, fallback, circuit breaking
Budget Amnesia, where a budget lives in an object shorter-lived than the scope it bounds so every check passes and the total is unbounded, and the Second Plant: a fallback model is a different process every bound was never designed against.
Gated execution
Consequence Gating: a gate cannot decide whether an action is correct, because nobody in the system can. It decides whether the system is allowed to be wrong about it — what being wrong would cost, and who is there to carry it.
Deterministic constraint systems
The Expressible Set of actions a model can propose at all — the cheapest safety work is removing actions from it — and Split Declaration, a fact about a tool recorded in two artifacts that never fail when they diverge.
The gate had nothing to refuse
Why prompt injection passes gates that check actions, and the two defenses that hold: the Frozen Plan, fixed before the first untrusted span enters the window, and the Copied Argument, an argument value provably present in untrusted text before the model produced it.
State, checkpoints, resume
The Restore Line, the declared field-by-field split between what a checkpoint restores and what a resumed run re-derives, and Replay Debt: the side effects a node already performed, which a resume performs again.
Cost, depth, width
The Receipt Ceiling, a limit checked against spend already incurred that fires only after the overrun, and the Width Gap: a depth limit counts steps and is blind to how many paths one step opens.
Authority bands
Authority Bands earned by the reversibility and blast radius of actions rather than by model quality, Earned Width — good behaviour may widen how much a run does inside its band, never move it across one — and the composite band no single run can see.
Testing the harness without the model, and with a bad one
The Hostile Double, a model substitute drawn from the measured failure distribution and your incident log rather than the test author’s imagination, and the Uncaught Set of behaviours no control claims to catch.
Attributing failures to a layer
The Exoneration Asymmetry — a layer is convicted by a finding but cleared only by a record written at the time — and the Oracle Deficit that every substitute for a test-suite verdict carries.
Failure modes of the control layer itself
The Silent Pass, where a dead control produces exactly the output of a live one on traffic that does not need it, and the Stop Path: two stops that both depend on the harness running are one stop.
The framework seam
The Second Executor that appears when framework nodes call controls directly and silently drop part of their discipline, and what deleting the framework does and does not prove about a harness.
Parts still marked in progress are covered in full by the book today — the chapter ranges above are exact. The articles are being written against those chapters.
The companion code
The companion repo builds the harness for one billing support agent, one git tag per chapter. Every tag passes its test suite with no API key, no network, and no model — the harness is tested as a control system, not as a prompt.
View the companion repository on GitHub →Other guide series
GenAI Architectures
There are eight architectures and seven boundaries between them. Every boundary is a yes-or-no question about the task — not a judgment call about the design.
Chat Templates
A chat template is the contract between your messages and the exact token stream the model was trained on. Get it wrong and nothing raises — the model just gets quietly worse.
Agentic AI with LangGraph
An eight-part guide to the part of agent engineering nobody demos: making the thing survive a restart, a concurrent write, a bad tool call, and a Tuesday deploy.
Harness Engineering
AI systems do not fail at the model layer. They fail in the code around it — and that code is a designable layer with its own architecture.
RAG Engineering
Most RAG failures happen upstream of the LLM, accumulate invisibly, and only surface as an incident. This series names each one.
Claude Code Playbook
Claude Code has four layers — context, skills, subagents, hooks — and the variable that decides which one you need is enforceability, not capability.
AI Control Plane
A fleet of agents needs a layer that is not itself an agent — observability, policy, orchestration, versioning, cost, and audit, built as infrastructure.
Agentic AI in the Data Center
In physical infrastructure, an agent’s autonomy is capped by reversibility and jurisdiction — not by how good the model is.
Codebase to Architecture Doc
A three-pass methodology for compressing any codebase into validated diagrams and decisions — and then a LangGraph pipeline that runs it for you.
Federated AI Agents
Every call that crosses from one owner to another is a seam, and a seam has five properties — contract, authority, state, failure, and accountability — that are free inside one deployment and have to be built across it.
