Guides · Series · 3 parts
Evaluating LangGraph Agents in Production
Every layer of an agent eval stack is a measuring instrument with its own error rate — the grader included — and none of them should gate a deploy until it has been scored against known faults.
The eval suite is an instrument too
A refund agent reports “your refund of 1,499 has been processed.” The eval case passes. The eligibility check never ran and the refund table has no row. Nothing about the model failed in a way the harness could see, because the harness graded the one artifact the agent writes about itself.
The usual fix — grade the trajectory and the end state as well — is right, and it is not the end of the problem. Trajectory checks can be too loose and let a broken agent through, or too strict and fail a correct agent that took a different valid path. Checkpoint replays can compare the wrong worlds. LLM graders can believe the transcript. Each of those is an error rate, and each can be measured.
This series measures them, on one refund-processing agent that grows across all three parts, with deliberately seeded faults as the ruler. Everything runs locally on open-weight models: no tracing platform, no API keys.
Where this picks up
The LangGraph series ended its evaluation part by mapping what agent evals assert on: what the model said, what the world shows, and what the runtime kept. This series takes that map as given and asks the next question — how often each layer of the eval stack is wrong, and what it costs to find out.
Which part solves my problem
The series reads in order, but if something is already broken, start here.
- “My eval suite is green and I still do not trust it.”Part 1 →
- “I tightened the trajectory checks and now correct runs fail.”Part 1 →
- “A one-line prompt change costs a full eval rerun and I cannot attribute the score change.”Part 2 →
- “My LLM judge scored a run 5/5 that never actually did the thing.”Part 3 →
- “I need a CI gate for agent quality that survives nondeterminism.”Part 3 →
The series
0 of 3 published
Your Agent Passed the Test and Still Failed the Task
An answer-only harness is blind by a measurable amount, and that amount belongs to the harness, not the agent. Measure it with seeded behavioural faults — and measure the brittle fails that stricter trajectory checks trade it for.
The Checkpointer Is Already Your Eval Harness
A checkpoint fork turns a full rerun into a paired comparison, which is where the cost and variance savings come from. It also restores the thread and not the world, which is where forked evals quietly go wrong.
A Grader You Haven’t Calibrated Is a Random Number Generator with a Rubric
A grader is a harness too. Score it against the same seeded faults, require it to cite evidence for every score, and measure its agreement with human labels before its number gates anything.
Other guide series
GenAI Architectures
There are eight architectures and seven boundaries between them. Every boundary is a yes-or-no question about the task — not a judgment call about the design.
Agentic AI with LangGraph
An eight-part guide to the part of agent engineering nobody demos: making the thing survive a restart, a concurrent write, a bad tool call, and a Tuesday deploy.
Harness Engineering
AI systems do not fail at the model layer. They fail in the code around it — and that code is a designable layer with its own architecture.
RAG Engineering
Most RAG failures happen upstream of the LLM, accumulate invisibly, and only surface as an incident. This series names each one.
Claude Code Playbook
Claude Code has four layers — context, skills, subagents, hooks — and the variable that decides which one you need is enforceability, not capability.
AI Control Plane
A fleet of agents needs a layer that is not itself an agent — observability, policy, orchestration, versioning, cost, and audit, built as infrastructure.
Agentic AI in the Data Center
In physical infrastructure, an agent’s autonomy is capped by reversibility and jurisdiction — not by how good the model is.
Codebase to Architecture Doc
A three-pass methodology for compressing any codebase into validated diagrams and decisions — and then a LangGraph pipeline that runs it for you.
