Guides · Series · 8 parts
Harness Engineering
AI systems do not fail at the model layer. They fail in the code around it — and that code is a designable layer with its own architecture.
The failures are not where you are looking
When an LLM feature misbehaves in production, the instinct is to reach for the model: a better prompt, a bigger model, a different provider. That instinct is usually wrong. The model produced a plausible token sequence, which is the only thing it ever promised to do. What broke was everything around it — the code that decided what the model saw, what it was allowed to do with the response, and what happened when the response was malformed.
That surrounding code is the harness, and most teams do not think of it as a layer at all. It accretes: a try/except here, a JSON repair there, a retry someone added after an incident. Nobody designed it, so nobody can reason about it, and each new failure adds another patch to a structure that has no shape.
This series gives that structure a shape. Each part takes one layer of the harness, explains the specific failure it exists to absorb, and shows what it looks like built deliberately rather than accreted.
Seven layers, one argument
The layers stack in the order a request travels. Input arrives and must be normalized before the model sees it. Context is assembled — what the model sees is what the model does, so the window is managed infrastructure, not a dump bucket. The model proposes an action, and gated execution decides whether the system actually performs it, because valid output is not safe output. The response is validated and repaired. Failures are absorbed by retry, fallback, and circuit breaking, because your provider will have an incident. Long-running work is checkpointed so the agent does not start over. And the tool registry constrains what the model can even attempt, closing the gap between the capabilities it imagines and the capabilities it has.
None of these layers are exotic. Every one of them is standard practice in ordinary distributed systems engineering. The argument of this series is that LLM systems do not get an exemption just because the unreliable component happens to be a model.
Which part solves my problem
The series reads in order, but if something is already broken, start here.
- “My LLM feature is unreliable and I keep tuning the prompt instead of the system.”Part 1 →
- “Users can paste anything into my system and something eventually breaks.”Part 2 →
- “The model ignores the one document that mattered because it was buried mid-context.”Part 3 →
- “My agent took a real action it should have asked about first.”Part 4 →
- “The model returns malformed JSON often enough that downstream parsing is a minefield.”Part 5 →
- “My provider had an outage and my whole system went down with it.”Part 6 →
- “A long task fails at step 40 and restarts from step 1.”Part 7 →
- “The model invents tools and calls APIs with parameters that do not exist.”Part 8 →
The series
8 of 8 published
Harness Engineering: The Missing Layer Between LLMs and Production Systems
Why AI systems don't fail at the model layer - and how designing the right execution harness turns brittle prompts into reliable infrastructure
Normalization and Input Defense: Hardening the Entry Point of Your LLM System
Every unreliable LLM system has a porous entry point. Here's how to build the layer that ensures the model only ever sees clean, controlled, safe input.
Context Engineering: What the Model Sees Is What the Model Does
The Lost in the Middle problem isn't a model bug. It's a context design failure - and fixing it requires treating the context window as managed infrastructure, not a dump bucket.
Gated Execution: Why Your Agent Should Never Act Without Permission
Valid output is not safe output. The Gated Execution layer is the firewall between what the model proposes and what the system actually does - and it's the difference between an agent that assists and one that causes incidents.
Validation Layer Design: Building the Reflex That Catches What the Model Gets Wrong
The model will produce malformed output. Not occasionally - regularly. The Validation Layer is the only thing standing between that malformed output and your downstream systems.
Retry, Fallback, and Circuit Breaking: Building LLM Infrastructure That Survives Outages
Your LLM provider will have an incident. The question is not whether your system fails when that happens - it's whether you designed for it beforehand.
State Management for Agentic Systems: How to Build Agents That Don't Start Over
A long-running agent without state management is a gamble. You're betting the entire task completes before something goes wrong. At production scale, that bet loses constantly.
Deterministic Constraint Systems: Building Tool Registries That Keep Agents in Scope
The model will try to use tools it doesn't have. It will call APIs with parameters that don't exist. It will invent capabilities. The constraint system is how you make the gap between what the model thinks it can do and what it can actually do exactly zero.
Other guide series
Agentic AI with LangGraph
An eight-part guide to the part of agent engineering nobody demos: making the thing survive a restart, a concurrent write, a bad tool call, and a Tuesday deploy.
RAG Engineering
Most RAG failures happen upstream of the LLM, accumulate invisibly, and only surface as an incident. This series names each one.
Claude Code Playbook
Claude Code has four layers — context, skills, subagents, hooks — and the variable that decides which one you need is enforceability, not capability.
AI Control Plane
A fleet of agents needs a layer that is not itself an agent — observability, policy, orchestration, versioning, cost, and audit, built as infrastructure.
Agentic AI in the Data Center
In physical infrastructure, an agent’s autonomy is capped by reversibility and jurisdiction — not by how good the model is.
Codebase to Architecture Doc
A three-pass methodology for compressing any codebase into validated diagrams and decisions — and then a LangGraph pipeline that runs it for you.
