LangGraph Evals Test the Answer, Not the Thread

One customer, one ticket, one approval decision, two gates, three charges. Here is the transcript, printed as the side effects fired.

invoke      -> interrupts pending: 1               side effects so far: ['charge:8812']resume #1   -> interrupts pending: 1               side effects so far: ['charge:8812', 'charge:8812']resume #2   -> done=True approvals=['yes', 'yes']               side effects so far: ['charge:8812', 'charge:8812', 'charge:8812']TOTAL side effects: 3  (correct answer: 1)final output      : approvals=['yes', 'yes'] done=True

The node is an ordinary human-in-the-loop approval gate with two interrupt() calls and one irreversible effect at the top of its body. Nothing crashed. Nobody retried. Here is what a LangGraph eval would have checked:

What an output assertion checks:  assert result['done'] is True                  -> True  assert result['approvals'] == ['yes','yes']    -> TrueBoth pass. The duplicated charge is not in the output.

The final answer is correct. Both assertions pass. The approvals list has exactly the two entries it should have, in the right order, and the run completed. Nothing about the returned object is wrong.

This is not an obscure edge case that I constructed to make a point. It is langgraph issue #6208, opened on 2025-09-26 by Caspar Broekhuizen, a LangGraph maintainer. Part 4 named it in its list of reasons nodes re-run, including the detail that a fix was merged on 6 October 2025 and reverted two days later. This part reproduces it, ten months on, still open. The title is the whole bug: "Do not re-execute a node that interrupted unless all of its interrupts have been resumed." The maintainer's own summary is that "a node with two interrupts will rerun after only one resume." That issue is cited by an independent security paper as cross-framework evidence, which I will get to.

The part I care about is not the bug. It is that the run produced a correct-looking answer while doing the wrong thing three times, and that no assertion anyone writes about an agent would have caught it.

The thesis: a thread holds three kinds of state and evaluation asserts on two

A LangGraph thread accumulates three distinct kinds of state, and the shipped evaluation stack asserts on one of them. Dimension two is claimed, and one research system does it properly, but no harness in my survey reaches it either.

What the model said. The output, the trajectory, the tool-call sequence, the final message. This is what output assertions, trajectory matchers, and LLM-as-judge graders read. It is dimension one, and it is where roughly all agent evaluation work lives.

What the world shows. State your application owns, sitting outside anything a checkpoint restore can reach: the row in the payments table, the file on disk, the ticket status in the system of record. This is dimension two. It is real, it is asserted on by good teams by hand, and there is published work doing it properly.

What the runtime kept. The checkpoint. The step counter, the re-derived step allowance, the pending writes, the interrupt records, the grant identifiers, the middleware counters, the message transcript that compaction removed from the view but not from storage. This is dimension three, and it is asserted on by nobody.

Dimension three is not defined by where the bytes live. A checkpoint is a row in a table, and this article queries it with ordinary SQL. It is defined by the rollback surface: dimension three is inside it, dimension two is outside it, and that is the whole discriminator. Who declared the schema is usually the runtime but is not what decides it - your own messages channel and Part 4's in-state counter are both dimension three, because a restore reaches them. That distinction is what makes the remedy at the end of this article work, because the remedy is precisely to move a record out of dimension three and into dimension two.

Three named defects from this series live in dimension three: Part 4's re-granted step allowance, Part 5's recoverable transcript, and the Halfway pattern counter from Parts 4 and 6. None of those changes the answer. Each changes what the runtime is holding when the next entry starts.

I am quantifying over those three defects and not over parts, because several other findings in this series do not fit and a reader will check. Part 2's Order-Coupled State needs two writers to a non-commutative key in one superstep, which is one entry, and it changes the state value. Part 3's ungated refund fired on turn one and printed REFUNDED 4999 on 8812. Part 6's Width-Blind Bound fires on a cold thread with no checkpointer at all. So does Part 6's per-subgraph allowance re-grant, which is the one I most wanted to claim here and cannot: Part 6 measured it across a compile boundary rather than a resume, and said so in those words.

Those are first-entry defects, and naming them sharpens the point rather than weakening it. A first-entry width blowup and a second-entry allowance re-grant are different classes, and the same eval suite misses both for different reasons. It misses the width blowup because nobody asserted on cost. It misses the allowance re-grant because the condition never existed during the test. This article is about the second kind, and it is a smaller set than I first wrote down.

Part 1's thesis was that durability moves lost state and no recovery, while hallucinated control flow does not move. Parts 2 and 3 are that unmoved half. Parts 4 through 6 are the half durability created.

And there is a structural reason the eval stack cannot see the second group. Each of those three defects requires prior persisted state to have been carried into the run. That state cannot be there on a thread's first entry, because there is no earlier entry to have written it, so there is no bound to re-derive, nothing to restore, and nothing to re-arm.

I am going to name the thing that hides them: the First-Entry Assumption. It is the assumption, built into every agent evaluation harness I surveyed, that the run under test is the system's first entry into its own persisted state. I have defined it framework-neutrally on purpose, following Part 2's lesson that a term defined without reference to one implementation outlives a change to that implementation. Nothing in the definition is about LangGraph. LangGraph is just where I can measure it.

The consensus belief I am arguing with is not "evals are useless." It is much narrower and much more widely held: that a green agent eval suite tells you something about how the agent behaves in production. For the three defects named above, it tells you nothing at all, because the suite never creates the condition under which they exist.

Prior art: crash recovery bugs, crash consistency, and ACRFence

Before going further I have to give away most of the novelty, because the honest version of this argument is stronger than the overclaiming version.

A defect that only appears after re-entry into persisted state is decades old and has at least two established names.

In software dependability it is a crash recovery bug. Gao, Dou, Qin and colleagues published an empirical study of them at ESEC/FSE 2018, covering 103 bugs across ZooKeeper, Hadoop MapReduce, Cassandra and HBase. Their root-cause breakdown is worth reading beside this article: incorrect state recovery 28 percent, concurrency 21 percent, incorrect crash or reboot detection 18 percent, incorrect backup 17 percent, incorrect state identification 16 percent. Eighty-seven percent were triggered by three or fewer crashes and at most one reboot, and 12 percent of the fixes were themselves incomplete.

In storage and filesystems it is a crash-consistency bug, with a taxonomy covering atomicity, ordering, unpersisted data and failure recovery, and with real tooling to find them. CrashMonkey and its bounded black-box testing work, WITCHER for persistent memory, and later crash-injection frameworks all exist to drive a system through a crash and then validate the state it recovers into.

So the class is named, studied, and has purpose-built tooling in adjacent fields. Three things separate what I am describing from that prior work, and I want them stated before any of the measurements rather than defended after them. Two of the three are Part 4's, restated because the argument here does not stand without them. Only the third belongs to this part.

First, no crash is required. Both the crash-recovery and crash-consistency literatures define the class by a crash or a reboot. Part 4's Self-Restoring Bound fires on an ordinary second turn on a healthy process. So does the transcript at the top of this article, which contains no crash, no restart, and no failure of any kind. That is a genuine widening of the trigger condition, and it matters practically: crash-injection tooling would not find these, because there is no crash to inject.

Second, the corrupted thing is the runtime's own bookkeeping, not the application's data. Crash-consistency bugs are about your file being half-written. These are about the framework's step counter, its interrupt records, its allowance derivation. Application-level invariant checks do not cover them because they are not application state.

Third, and this is the actual contribution, the tooling asymmetry runs the wrong way. Distributed systems have crash-injection harnesses. Filesystems have crash-consistency checkers. Agent evaluation has nothing equivalent, and the discipline it does have actively removes the precondition.

I also have to concede a paper that gets closer than any of those. ACRFence (Zheng, Yang, Zhang and Quinn, arXiv:2603.20625, 2026-03-21) coins "semantic rollback attacks" and names two classes that map onto this series almost one to one. Action Replay: a malicious payee who controls one service in the tool chain deliberately triggers a crash after a payment succeeds, the framework restores, and the agent re-issues the payment with a fresh reference identifier. Authority Resurrection: an attacker restores the agent to the state just after approval was granted, so "the agent now holds the token but has no memory of having used it." That second one is Part 5's re-armed grant with an adversary attached to it.

ACRFence also states the near-invisibility property directly. The attacks are "difficult to audit" because "each transaction carries a distinct reference and appears legitimate in isolation." That is uncomfortably close to my point, and Part 4 already conceded this paper once.

What it does not own: its threat model is adversarial and needs a crash. Mine needs neither. Its experiments ran on Claude Code CLI with Qwen3-32B and Model Context Protocol (MCP) tool servers, not on LangGraph. The paper also states that its evaluation "validates the attacks but does not yet include an implementation of ACRFence itself." Most importantly, it says nothing at all about evaluation harnesses. It names two attack classes. It does not name a testing blind spot.

One more concession, because it is the one people will reach for. AgentAtlas (Mazaheri and Mazaheri, arXiv:2605.20530, 2026-05) audits 15 agent benchmarks on six coverage axes, and one of those axes is Memory and state. It also has a trajectory-failure category 9, "State or memory contamination," defined as stale or contaminated context distorting later decisions. So it is false to say no evaluation taxonomy has a state axis, and I am not going to say it.

Two more papers sit close enough that I would rather name them than have someone else name them for me. If I skipped them, a reader who knows them would stop trusting the rest of this.

Observability for Delegated Execution in Agentic AI Systems (Mishra and Sharad, arXiv:2606.09692, 2026-06) ran a LangGraph micro-deployment instead of only citing one, which is more than most adjacent papers manage. It defines a delegation-observable system and coins Authority-Causality Divergence. But the checkpoint is not telemetry, and the paper never mentions durable state or resumption.

Revisable by Design (arXiv:2604.23283, 2026-04) classifies agent actions as idempotent, reversible, compensable or irreversible, and concludes that an agent's flexibility is bounded by its reversibility. Its rollback is semantic. Mine is a runtime restoring bytes it wrote earlier.

None of this is unique to agents, either. Retrieval evaluation has its own long-standing version of a metric that stays green while the failure that matters is invisible to it, which I have written about as the evals blind spot. The pattern generalises. What is specific here is which state the blind spot covers.

Back to AgentAtlas, where I want to use their number rather than dispute their axis. On the Memory and state axis, exactly 1 of 15 benchmarks scores strong: ToolSandbox. Ten score partial. Four score it absent outright, including WebVoyager, GAIA and AssistantBench. And the axis itself is about context, which is a model-behaviour failure observed through the trajectory. AgentAtlas category 9 is contamination that changes what the model does. A second-entry defect is runtime bookkeeping that does not change what the model does at all. The charge fired three times and the model's contribution to the transcript was identical.

Why durable agents broke agent evaluation in 2026

Two things changed.

Durable execution stopped being exotic in the agent world. Checkpointers are the default recommendation for anything with a human approval step, and this series has spent six parts arguing they are the right call. Every part of that argument still holds. But durability is precisely the property that makes a second entry possible, so the same feature that fixed lost state created the condition for a defect class that first-entry testing cannot reach. That is not an argument against durability. It is an argument that the testing discipline did not move when the runtime did.

At the same time, agent evaluation professionalised fast, and it professionalised around a unit of work that is one run over one dataset item. The datasets and the graders got better, and the continuous-integration story got dramatically better. AEVAL (arXiv:2607.16345, 2026-07) is the best example I found: deterministic change-triggered evaluation, per-skill contracts, executor and grader separation with a first-attempt grading rule, and grounded fix suggestions posted as inline merge-request comments. It formalises "self-correction bias," the effect where a shared identity between the agent and its own assertions converts real failures into spurious 100 percent pass rates.

AEVAL is genuinely excellent work, and it runs each evaluation case in an isolated session. It contains no mention of persistence, checkpoints, durable state, resume, or LangGraph. That is the fairest way I can put the situation. The best-in-class version of this discipline is structurally blind to the defect class. Not through carelessness. Through a design choice that is correct for everything else it does.

The wrong way: how the eval stack actually compiles your graph

I opened each harness. The nuances matter more than the headline.

Three of those rows need honest treatment, because the naive version of this table is wrong.

Inspect AI does build a checkpointer, and this is the most interesting row. Its documentation says evaluations running for hours or even days can use checkpointing for resilience against unexpected termination. It periodically persists the state of each sample, so an evaluation can resume mid-sample from its most recent checkpoint. What it saves is substantial: the current state of the main agent including messages and compaction, the sandbox filesystem, and the store and event history of the sample.

So a major evaluation framework ships durable execution, resumes from it, and never once opens what it wrote to check whether the contents are right. There is no checkpoint validation, no assertion on contents, no replay of old state against new agent code. It is durability of the harness, and the harness treats its own persisted state as opaque. That is the strongest version of my point, not a counterexample to it.

agentevals reads a checkpointer. It exposes extract_langgraph_trajectory_from_thread(), which reaches into a thread and extracts inputs, results and steps for a graph-trajectory evaluator. It uses the checkpointer as a transcript source. Budgets, counters, pending writes and grant ledgers are not in the extracted shape, so the state that carries the defect never reaches the evaluator.

LangSmith Multi-turn Evals was announced 2025-10-23, not this year, and it is online-only. It scores completed production threads after an idle timeout, assembling messages from ingested traces with deduplication. It does not create the thread and it does not drive the agent. The 2026 additions are thread-evaluator tooling, still scoring traces rather than asserting on state.

That last point kills the easy version of my original claim, and I want to be explicit that it did. Before the research I believed no eval harness ever enters a thread twice. That is false. Multi-turn threads are exercised. The surviving claim is narrower and, I think, worse: the path is exercised and the assertion surface still cannot see the defect, because the harm never reaches the answer.

What the harness can actually reach, measured

The LangSmith page on evaluating a LangGraph application compiles the graph like this, with no memory:

app = workflow.compile()

I wanted to know what that costs you in assertion power, so I built the same graph twice: once as the eval harness compiles it, once as production compiles it.

# harness 1 - the eval harness, exactly as the docs compile itapp = build().compile()out_eval = app.invoke({"customer_id": "c-1"}, config={"recursion_limit": 8})# can this harness assert on runtime state at all?try:    snap = app.get_state({"configurable": {"thread_id": "anything"}})except Exception as e:    print(f"  get_state raised {type(e).__name__}: {e}")try:    hist = list(app.get_state_history({"configurable": {"thread_id": "anything"}}))except Exception as e:    print(f"  get_state_history raised {type(e).__name__}: {e}")

returned output : {'customer_id': 'c-1', 'evidence': ['doc-for-c-1'], 'answer': 'resolved for c-1'}can the eval assert on runtime state?  get_state raised ValueError: No checkpointer set  get_state_history raised ValueError: No checkpointer set

Be precise here. The eval harness does not fail to inspect the checkpoint. It has no object to inspect. get_state and get_state_history are the two doors into dimension three, and both raise ValueError: No checkpointer set when the graph was compiled the way the evaluation documentation compiles it. Dimension three is unreachable by construction.

Then I ran the same graph the way production runs it: with a checkpointer, twice, on one thread.

entry 1 output   : {'customer_id': 'c-1', 'evidence': ['doc-for-c-1'], 'answer': 'resolved for c-1'}entry 2 output   : {'customer_id': 'c-1', 'evidence': ['doc-for-c-1', 'doc-for-c-1'], 'answer': 'resolved for c-1'}--- what an OUTPUT assertion sees ---  answer identical across entries : True  assert out['answer'].startswith('resolved') passes on entry 1: True  ... and on entry 2                                          : True--- what the CHECKPOINT shows ---  langgraph_step  at 'gather':  entry1=1  entry2=5   -> counter CONTINUED  evidence length at 'gather':  entry1=0  entry2=1   -> state DID accumulate

The answer field is byte-identical on both entries. The thread underneath it is not: the step counter has moved from 1 to 5, and the evidence list the second entry started from already had an item in it. An assertion on answer cannot distinguish a cold thread from a warm one. That is the whole shape of the problem in four lines of output.

Why remaining_steps read None, and what that nearly cost me

The first version of that probe tried to prove the point by reading remaining_steps on both entries and comparing them. It printed allowance RE-DERIVED (same), which looked like a clean confirmation.

It was comparing None to None. The managed value was not populated under the state schema I had subclassed, so the comparison was vacuous and the reassuring output was meaningless. I only caught it because the value looked suspiciously round.

I am including this because it is the same failure mode the article is about, one level up. An assertion that compares two absent values passes, and a passing assertion reads as evidence. The fix was to stop reading a managed value and start counting executions, which is the next section.

What agent evals assert on, and what they miss

flowchart TD
    RUN["One entry into a LangGraph thread"] --> D1["Dimension 1<br/>What the model SAID<br/>output, trajectory, tool calls"]
    RUN --> D2["Dimension 2<br/>What the WORLD shows<br/>payments row, ticket status, file"]
    RUN --> D3["Dimension 3<br/>What the RUNTIME KEPT<br/>step counter, allowance, pending writes,<br/>interrupt records, grants, transcript"]

    D1 --> A1["Asserted by:<br/>output matchers, trajectory<br/>evaluators, LLM-as-judge"]
    D2 --> A2["Asserted by:<br/>good teams, by hand<br/>no surveyed harness"]
    D3 --> A3["Asserted by:<br/>no surveyed harness"]

    A3 --> HIDE["First-Entry Assumption:<br/>no second entry in the test,<br/>so nothing to re-derive from"]

    style RUN fill:#4A90E2,color:#FFFFFF
    style D1 fill:#98D8C8,color:#2C2C2A
    style D2 fill:#98D8C8,color:#2C2C2A
    style D3 fill:#FFD93D,color:#2C2C2A
    style A1 fill:#6BCF7F,color:#2C2C2A
    style A2 fill:#6BCF7F,color:#2C2C2A
    style A3 fill:#E74C3C,color:#FFFFFF
    style HIDE fill:#C2185B,color:#FFFFFF

Dimension two is not hypothetical and I do not want to leave the impression that nobody works on it. GroundEval (Flynt, arXiv:2606.22737, 2026-07) evaluates against ground-truth state transitions and verifiable state facts rather than model preference, deterministically. It is exactly the right way to do dimension two. It also explicitly does not involve runtime checkpoints or resumption, which is what makes it a clean illustration of the split rather than a counterexample.

There is a smaller version of this split inside LangChain's own documentation, and it is the cleanest internal evidence I found. The testing page uses a checkpointer, a thread_id, update_state(as_node=...) and interrupt_after. The evaluation page uses none of them. It also carries a line of prose about optionally passing the memory when compiling the graph, directly above code that compiles with no memory at all. The two pages disagree about whether a graph under test has a thread. The evaluation page disagrees with itself.

The First-Entry Assumption, defined properly

I want to be precise about what the term names, because a vague version of it would be useless as a citation handle.

A harness makes the First-Entry Assumption when the run under test is the system's first entry into its own persisted state. That is the whole definition. Three properties explain why it survives contact with careful engineers, and none of them is a condition on the definition.

It is invisible from inside a passing suite. The assumption is not a setting anyone chose. Nobody wrote first_entry_only = True. It is the residue of two ordinary lines: compiling a graph without a checkpointer, and allocating a fresh thread_id per case. Neither looks like a decision about coverage when you read the test file.

It is correct for everything else the suite measures. This is the property that keeps it alive. Isolation genuinely does prevent correlated failures and genuinely does stop score inflation. A harness that abandoned it would get worse at its main job. So the assumption is not a bug to be fixed but a trade-off nobody was told they were making.

It removes the precondition rather than failing to check it. This is the part that separates it from ordinary missing coverage. A gap in coverage means the test could have caught the defect and did not look. Here the defect cannot exist during the test, because the state it needs was never carried in. You could add assertions on every field of the returned object, forever, and change nothing.

That third property is why I think the term earns its place. The usual remedy for a blind spot is a better assertion. There is no assertion that fixes this one, because the run under test is not the run that has the problem.

A concrete test for whether your harness has it. Take any evaluation case that touches an irreversible effect. Ask one question: could this case fail differently if it ran a second time on the same thread_id? If the answer is no, ask why not. If the reason is that the harness never runs anything a second time on the same thread_id, you have found it.

The distinction I would like to survive this article is between two things that sound the same. Untested means the assertion was not written. Untestable under the current discipline means the assertion cannot be written, because the harness does not build the world the assertion would be about. Dimension three is the second one, and it stays the second one no matter how good your graders get.

The second entry re-grants the allowance

Part 4 measured a step budget arriving back at full on every entry. Here it is again in this part's units: what one eval run observes, against what production grants on one warm thread.

The graph never terminates. It is a single node with an edge back to itself, so the runtime has to stop it.

LIMIT = 8def spin(state: LoopState) -> dict:    EXEC.append(get_config()["metadata"].get("langgraph_step"))    return {"ticks": [1]}g = StateGraph(LoopState)g.add_node("spin", spin)g.add_edge(START, "spin")g.add_edge("spin", "spin")   # never terminates

First, the eval harness: one run, no checkpointer.

B - the eval harness: ONE run, no checkpointer  halted: GraphRecursionError  message: Recursion limit of 8 reached without hitting a stop condition...  node executions observed by the eval: 8  langgraph_step values: [1, 2, 3, 4, 5, 6, 7, 8]

Eight executions, one clean GraphRecursionError, exactly as configured. If your eval asserts that a runaway graph is bounded at 8, it passes, and the assertion is true about what it measured.

Now production: three ordinary entries on one warm thread, same limit, same config object.

C - production: THREE entries on ONE warm thread, same recursion_limit  entry 1:   8 node executions   langgraph_step 1 -> 8  entry 2:   8 node executions   langgraph_step 11 -> 18  entry 3:   8 node executions   langgraph_step 21 -> 28  eval harness measured        : 8 executions, once  production, per entry        : [8, 8, 8]  production, total across 3   : 24  ticks accumulated in state   : 24

Twenty-four executions against a limit of eight, and the count in state agrees independently at 24.

Two details keep this consistent with what earlier parts measured, and I checked both against their published text rather than trusting my memory of them.

The step counter continues while the budget resets. It runs 1 to 8, then 11 to 18, then 21 to 28. That is Part 4's finding exactly: the counter continues from the checkpoint, and what resets is the allowance. Part 4 corrected the inference readers drew from Part 1 here, which Part 3 had forwarded as max_steps resetting to zero. Part 4 was explicit that Part 1's own sentence about a hand-rolled loop is right and that what it wanted to kill was the inference. I am restating the corrected version.

The gap of 2 between entries is the entry cost Part 4 measured, spent on applying input and entering the loop. Part 4's transcript reported step values of 8, 18, 28 and 38 across four entries at the same limit, which is the same 10-tick spacing. My entries pass an input dict rather than None, which is why the node-execution count per entry is 8 rather than 10. Same mechanism, different accounting.

Part 6's identity holds where Part 6 said it holds and breaks where Part 6 said it breaks. On a cold invocation, remaining_steps reads 7 at step 1 under a limit of 8, so langgraph_step + remaining_steps == recursion_limit. Part 6 stated plainly that this is per-invocation and does not survive a resume. Entry two above is that sentence in transcript form: langgraph_step alone reaches 18 under a limit of 8.

Nothing here is broken. The bound fired every single time. It fired against a quantity that resets, and the eval measured one entry.

Why an interrupted LangGraph node re-runs its whole body

The transcript at the top of this article deserves an explanation rather than an anecdote, so I read the installed package rather than the documentation or GitHub's main branch.

The documented replay rule says nodes before the checkpoint are not re-executed because their results are already saved, and nodes after it re-execute including their model calls, API requests and interrupts. It also says replay re-executes nodes rather than reading from a cache.

The shipped source is sharper, and the difference is the whole bug. In langgraph/pregel/_loop.py at 1.2.9:

def _reapply_writes_to_succeeded_nodes(self, tasks) -> None:    """Restore successful channel writes from checkpoint to in-memory tasks.    Skips control signals (ERROR, ERROR_SOURCE_NODE, INTERRUPT, RESUME)    so that failed/interrupted tasks remain with empty writes and will be    re-executed (or routed to error handlers) by the runner.    """    for tid, k, v in self.checkpoint_pending_writes:        if k in (ERROR, ERROR_SOURCE_NODE, INTERRUPT, RESUME):            continue        if task := tasks.get(tid):            task.writes.append((k, v))

So the rule is not "pending writes prevent re-execution." It is "successful channel writes prevent re-execution, and ERROR, INTERRUPT and RESUME writes are deliberately skipped so those tasks do re-execute." The runner executes only tasks whose writes are empty. An interrupted task is kept with empty writes on purpose. It therefore re-runs from the top of the body, side effects and all.

Part 4 already established the behaviour, and I am not going to pretend otherwise. It states plainly that interrupt() ignores the pending-writes rule, that the node restarts from the top on resume regardless of pending writes, and it quotes the human-in-the-loop documentation saying the runtime restarts the entire node from the beginning. A reader who took Part 4's word for it predicts the transcript at the top of this article correctly.

What the shipped source adds is the mechanism and one case nobody documents: control-signal writes are not writes for this purpose, which is exactly why an interrupted node behaves differently from a completed one.

There is a second carve-out immediately below it that was not in my research brief, and I only found it by reading past the function I came for:

def _resume_error_handlers_if_applicable(self) -> None:    """On resume, schedule error handlers for tasks that failed in a prior run.    Called right after ``_reapply_writes_to_succeeded_nodes`` during ``tick()``.    At that point, ``_reapply_writes_to_succeeded_nodes`` has already skipped    ERROR / ERROR_SOURCE_NODE writes, so a previously-failed task still has    empty ``writes``.  Without intervention the runner (which executes only    tasks where ``not t.writes``) would re-run the original node.    """

So the re-execution rule has three tiers, not two. A task that succeeded gets its writes restored and is skipped. A task that errored and has an error handler gets the handler scheduled instead of re-running. A task that errored or interrupted without a handler is re-executed from the top.

That third tier is where the transcript at the top of this article lives, and it is deliberate behaviour with a docstring explaining the intent.

Be careful about how far to push the docs-versus-source point, because the human-in-the-loop page is accurate about tier three and Part 4 already quoted it. The gap is narrower and sits in two places. The time-travel page's replay sentence describes tier one and does not mention that control-signal writes are exempt. And tier two is documented nowhere I could find. I want to flag one thing about tier two honestly: I read it from the docstring and did not run it, where tiers one and three both have transcripts in this article. Treat it as source-read, not measured.

flowchart TD
    RESUME["Resume into a thread<br/>runner executes only tasks<br/>whose writes are empty"] --> CHECK{"What did this task<br/>leave in the checkpoint?"}

    CHECK -->|"successful channel writes"| T1["Tier 1<br/>writes restored to the task<br/>NODE IS SKIPPED"]
    CHECK -->|"ERROR write, handler exists"| T2["Tier 2 (source-read,<br/>not measured here)<br/>error handler scheduled<br/>NODE IS SKIPPED"]
    CHECK -->|"INTERRUPT or RESUME write,<br/>or ERROR with no handler"| T3["Tier 3<br/>control signals skipped,<br/>writes stay empty<br/>NODE BODY RE-RUNS FROM THE TOP"]

    T3 --> EFFECT["Every side effect above the<br/>first interrupt() fires again"]
    EFFECT --> INVIS["Not recorded anywhere<br/>on the thread"]

    style RESUME fill:#4A90E2,color:#FFFFFF
    style CHECK fill:#7B68EE,color:#FFFFFF
    style T1 fill:#6BCF7F,color:#2C2C2A
    style T2 fill:#6BCF7F,color:#2C2C2A
    style T3 fill:#FFD93D,color:#2C2C2A
    style EFFECT fill:#E74C3C,color:#FFFFFF
    style INVIS fill:#C2185B,color:#FFFFFF

Tier two is the one worth knowing about in review. A node with an error handler and a node without one behave differently on resume, and nothing on the rendered graph says which is which.

The fork path charges again, and the final state hides it

Replay is not the only way back into a thread. Forking is the other, and it is a different code path, so I measured it rather than assuming the result carried over.

The correct way to fork is update_state() on a prior checkpoint's config, which returns a config you then invoke. I forked from the pre-interrupt checkpoint of a completed, healthy thread, on a node with a single approval gate.

A - normal run to completion on the original thread  body executions : 2  final state     : {'ticket_id': '8812', 'approvals': ['yes'], 'done': True}  checkpoints     : 3B - fork from the PRE-INTERRUPT checkpoint via update_state()  forking from step=0  body executions caused by update_state alone: 0  body executions caused by invoking the fork  : 1  body executions caused by resuming the fork  : 1C - what the thread records after the fork  TOTAL body executions across original + fork : 4  effects list  : ['charge:8812', 'charge:8812', 'charge:8812', 'charge:8812']  checkpoints visible on the thread now        : 5  final approvals in state                     : ['yes']

Four charges. Read the last line again: the final state holds approvals: ['yes'], which is exactly what a single clean run produces. One approval in the state, four charges against the customer.

Pull that transcript apart.

update_state() on its own executes nothing. The body executions happen when you invoke the config it returns. That is correct and documented behaviour. It does mean the cost of a fork is paid at a different call than the one that looks like it forks.

The fork stayed on the same thread_id. update_state() returns a config with a new checkpoint_id and the original thread identifier, so the forked history and the original history are interleaved on one thread. Checkpoints went from 3 to 5 and there is no field marking which two are the fork.

And the output assertion is worse off here than in the replay case. In the replay case the approvals list at least grew to ['yes', 'yes'], which a very suspicious reviewer might notice. After a fork the list is back to the length a correct run produces, because the fork rebuilt it from a checkpoint taken before the first approval landed. The only artifact whose value is wrong is the charge count, and the charge count is not on the thread. The checkpoint count did change, from 3 to 5, but nothing marks which two of those are the fork, and a number that merely moved is not a number you can write an assertion against.

This bears on langgraph #4987, which reports that after forking, graph.stream() reuses the original checkpoint identifier rather than minting new ones. It also bears on a Part 6 finding. Part 6 measured that checkpoint_ns is node:task_id and that the task id changes across a fork. An idempotency key built on it therefore makes a retry idempotent and charges a fork again. Part 6's sentence predicts this transcript. To be exact about what I ran: the probe used no idempotency key at all, it appended to a list, so it measures the re-execution such a key would have to survive rather than testing the key itself.

The rule, measured across four cases

I did not want to generalise from one transcript, so I measured the relationship between interrupt count and body executions for n = 0, 1, 2 and 3.

=== n_interrupts = 0 ===  invoke     : effects=1 pending_interrupts=0 snap.next=() snap.interrupts=0 done=True  FINAL      : body executions=1  done=True  checkpoints=3=== n_interrupts = 1 ===  invoke     : effects=1 pending_interrupts=1 snap.next=('review',) snap.interrupts=1 done=None  resume #1  : effects=2 pending_interrupts=0 snap.next=() snap.interrupts=0 done=True  FINAL      : body executions=2  done=True  checkpoints=3=== n_interrupts = 2 ===  invoke     : effects=1 pending_interrupts=1 snap.next=('review',) snap.interrupts=1 done=None  resume #1  : effects=2 pending_interrupts=1 snap.next=() snap.interrupts=1 done=None  resume #2  : effects=3 pending_interrupts=0 snap.next=() snap.interrupts=0 done=True  FINAL      : body executions=3  done=True  checkpoints=3=== n_interrupts = 3 ===  invoke     : effects=1 pending_interrupts=1 snap.next=('review',) snap.interrupts=1 done=None  resume #1  : effects=2 pending_interrupts=1 snap.next=() snap.interrupts=1 done=None  resume #2  : effects=3 pending_interrupts=1 snap.next=() snap.interrupts=1 done=None  resume #3  : effects=4 pending_interrupts=0 snap.next=() snap.interrupts=0 done=True  FINAL      : body executions=4  done=True  checkpoints=3interrupts | body execs | completed | checkpoints         0 |          1 |      True |           3         1 |          2 |      True |           3         2 |          3 |      True |           3         3 |          4 |      True |           3

The rule is exact: body executions equal interrupts plus one, for every value I measured, and every run completed successfully. A node with three approval gates runs its body four times. Any side effect above the first interrupt() call fires once per gate plus once more.

Note the last column. Three checkpoints, for every single case. One body execution and four body executions leave threads with the same number of checkpoints. Hold onto that, because it is the next section.

Why snapshot.next is the wrong signal to drive a resume loop

The first version of this measurement produced a different and confidently wrong table.

interrupts | body execs |  resumes | checkpoints | execs == interrupts+1         0 |          1 |        0 |           3 | True         1 |          2 |        1 |           3 | True         2 |          2 |        1 |           2 | False         3 |          2 |        1 |           2 | FalseRULE HOLDS FOR ALL MEASURED n: False

That probe drove its resume loop off snapshot.next, which is the obvious signal to use. It concluded the rule did not hold and that runs with more than one interrupt simply stop early.

It was wrong, and the reason is a finding in its own right. Look at the correct table above, at the n_interrupts = 2 row after the first resume:

resume #1  : effects=2 pending_interrupts=1 snap.next=() snap.interrupts=1 done=None

snapshot.next is the empty tuple while snapshot.interrupts still has one entry pending. From the first resume onward, .next reports nothing to do on a thread that is genuinely waiting for another approval. A driver that loops while .next is non-empty abandons the thread after the first resume, with done never set and the approval never collected.

snapshot.next is not a safe driver signal for a thread with interrupts. snapshot.interrupts is. I would not have found that if the first probe had agreed with the second. It took a third probe to work out which of the two was lying.

What a LangGraph checkpoint records about node re-execution

Having reproduced the defect, the obvious next move was to write the assertion that catches it. Read the checkpoint history, count how many times the node wrote, assert it is one. This is the thing no eval harness does, so it should be the payoff of the whole article.

def assert_node_ran_once(app, cfg, node: str) -> tuple[bool, int]:    """How many times did `node` actually execute? Count checkpoints    whose metadata records a write by this node."""    runs = 0    for snap in app.get_state_history(cfg):        writes = (snap.metadata or {}).get("writes") or {}        if node in writes:            runs += 1    return runs == 1, runs

It does not work.

  real side effects fired            : 3  checkpoint-recorded 'review' writes: 0  assert_node_ran_once('review')     : FAIL  per-checkpoint metadata (source=..., writes keys):    step= -1  source=input     writes=[]    step=  0  source=loop      writes=[]    step=  1  source=loop      writes=[]

Zero recorded writes for a node whose body ran three times. The function reports FAIL, which happens to be the right verdict for entirely the wrong reason: it would report FAIL on a perfectly healthy single execution too. It does not detect the defect it claims to detect. I discarded it rather than shipping it.

So I stopped assuming and dumped everything the thread holds, for the two-interrupt case and a one-interrupt control.

--- two-interrupt node (3 body executions) ---  checkpoints: 3    step= -1 source=input    next=('__start__',) writes=None n_tasks=1 n_interrupts=0    step=  0 source=loop     next=('review',)    writes=None n_tasks=1 n_interrupts=1    step=  1 source=loop     next=()             writes=None n_tasks=0 n_interrupts=0  distinct task_ids in writes table: 3--- one-interrupt node (2 body executions) ---  checkpoints: 3    step= -1 source=input    next=('__start__',) writes=None n_tasks=1 n_interrupts=0    step=  0 source=loop     next=('review',)    writes=None n_tasks=1 n_interrupts=1    step=  1 source=loop     next=()             writes=None n_tasks=0 n_interrupts=0  distinct task_ids in writes table: 3

Read those two blocks against each other. One node body ran three times. The other ran twice. The threads they left behind are identical on every artifact above: same checkpoint count, same step sequence, same source labels, same next values, same interrupt counts, same number of distinct task identifiers.

That last one is an aggregate, and an aggregate is exactly where a finding like this usually dies. Part 4 established that a resume value is persisted in pending writes under a reserved channel, so the obvious objection is that the raw writes table holds one row per resume and resumes + 1 reconstructs the count. I dumped every row for n = 1, 2 and 3 to find out.

=== n_interrupts=1  body executions=2 ===  total rows in writes table: 7    task=c6b4239a channel=__interrupt__    idx=-3  type=msgpack  vlen=84    task=00000000 channel=__resume__       idx=-4  type=msgpack  vlen=4    task=35544abf channel=__resume__       idx=-4  type=msgpack  vlen=5  __resume__ rows: 2=== n_interrupts=2  body executions=3 ===  total rows in writes table: 7  __resume__ rows: 2=== n_interrupts=3  body executions=4 ===  total rows in writes table: 7  __resume__ rows: 2  n=1: writes rows=7  __resume__ rows=2  body execs=2  n=2: writes rows=7  __resume__ rows=2  body execs=3  n=3: writes rows=7  __resume__ rows=2  body execs=4  Do the threads differ in the raw writes table? NO - identical     (the probe compares row count and __resume__ row count)

Seven rows every time. Two __resume__ rows every time, whether the node was resumed once or three times, so resumes + 1 gives 3 for all three threads while the real counts are 2, 3 and 4. The resume channel does not accumulate a row per resume. It holds the current one.

The only value that moves is the byte length of the approvals payload, and that is the graph's own application data. It is my list growing, not the runtime recording anything about execution.

Also note writes=None on every checkpoint. I nearly shipped that as a claim about interrupt threads, then realised I had only ever looked at interrupt threads, so I ran a control with two ordinary nodes and no gates at all.

--- NO interrupts, two ordinary nodes ---  step= -1 source=input    writes=None  step=  0 source=loop     writes=None  step=  1 source=loop     writes=None  step=  2 source=loop     writes=None--- ONE interrupt ---  step= -1 source=input    writes=None  step=  0 source=loop     writes=None  step=  1 source=loop     writes=None

None in both cases, on every checkpoint, so this is not something interrupts cause. I then ran the same control on InMemorySaver and got None on every checkpoint there too, which rules out the SQLite checkpointer being the cause. I did not test Postgres.

Any assertion built on metadata["writes"] is void here, including the one I wrote above. If you have example code that reads that field, it is either targeting a different version or it has never been exercised.

The execution count is not recoverable from the thread. Not from get_state_history(), not from checkpoint metadata, and not from the checkpointer's raw writes table either. Dimension three is not merely unasserted for this defect class. It is unrecorded.

Three limits on that claim, since it is the strongest one in the article. The metadata["writes"] is None control ran on both SqliteSaver and InMemorySaver. The raw writes table result is SqliteSaver only, because InMemorySaver has no such table to query, and I did not test Postgres on either. And it is a claim about the execution count specifically, not about everything in a checkpoint: plenty of dimension-three state is recorded perfectly well, which is why the budgets and counters from Parts 4 and 6 can be read even though this cannot.

That is a considerably darker finding than the one I set out to write, and it kills the clean version of the remedy. "Assert on the checkpoint" is not available, because the checkpoint does not know.

The right way: put the record where a restore cannot reach it

If the runtime does not record consumption, you have to. That is not a new recommendation in this series. It is Part 4's, and this measurement is the strongest argument for it that I have.

Part 4's rule was simple, and it is the same instinct as treating a checkpoint as a photograph rather than a live feed. Keep the grant identifier in state and the consumption record outside it, in a table the checkpointer does not own. Consume it with a compare-and-swap, not a read followed by a write. The reasoning there was about rollback: a restore returns everything inside the surface, so the spend record has to live outside it.

The measurement above adds a second reason that is about testability. A record outside the rollback surface is in dimension two, and dimension two is assertable. You cannot write a test that catches a duplicated charge by reading the thread, because the thread does not know. You can write one that reads the ledger, because the ledger is an ordinary table.

Be clear about what that costs, in Part 5's terms, because a Part 5 reader will raise it before finishing this section. Part 5's Retention Inversion has two halves, and this remedy deliberately uses one of them. Inside the rollback surface, deletions are impermanent. Outside it, writes are permanent, and no thread operation removes them. The ledger is on the permanent side by design. That is exactly what makes it assertable and exactly what makes it your retention problem: no delete_thread reaches it, it needs its own explicit policy, and the thread_id sitting in it is a link back to the transcript you may one day be compelled to destroy.

So Part 7's contribution to the remedy is narrow, and I will state it plainly: the eval layer cannot detect the ledger's absence. A system that skipped Part 4's advice entirely passes exactly the same eval suite as a system that took it. There is no assertion at the output level that distinguishes them, and there is no assertion at the checkpoint level either, because the information is not there.

That cuts both ways, and it is worth being honest about the circularity rather than leaving it for a reader to find. The test below can only run against a system that already built the ledger. If you never built one, there is no table to query and nothing to assert on, and the suite stays green exactly as before. The ledger is not only how you fix the defect, it is the only reason the defect becomes testable at all. That is an uncomfortable position - you have to have taken the advice before the test can tell you whether you took it - and I do not have a way around it. Nothing in the runtime records what you would need.

First, the second-entry test that does not work

The obvious fix, once you accept the problem, is to run each case twice on the same thread and check that the second run agrees with the first. I wrote that version before I wrote the working one. I am showing it because it looks correct and catches nothing.

# DO NOT SHIP THIS. It creates the second entry and then asserts on# the one thing that is guaranteed to be identical.def assert_rerun_is_stable(build_graph, saver, case) -> None:    app = build_graph().compile(checkpointer=saver)    cfg = {"configurable": {"thread_id": case.thread_id}}    first = app.invoke(case.inputs, config=cfg)    second = app.invoke(case.inputs, config=cfg)    assert first["answer"] == second["answer"]      # passes    assert second["done"] is True                   # passes

Both assertions pass on every transcript in this article. They pass on the thread that charged the customer three times, because done was True and the answer was correct. They pass after the fork, because the fork rebuilt approvals to exactly the length a healthy run produces.

This is the trap, and it is a specific one. Creating a second entry is necessary and not sufficient. If the test then asserts on the returned value, it has built the condition for the defect and looked away at the moment it mattered. The assert_node_ran_once() attempt earlier in this article failed the same way from the other side: it looked at the thread, and the thread did not know.

The second-entry test that does work

The working pattern has two halves and both are required.

Half one: the effect is recorded outside the rollback surface. This is Part 4's design, carried forward unchanged, with the table definition included so the assertion below has something real to read.

The table has to record two different things, and my first version recorded only one. It needs to know whether the effect fired, which is the guard. It also needs to know how many times the guard was asked, which is the execution count the checkpoint refuses to keep.

CREATE TABLE effects_log (    thread_id TEXT NOT NULL,    call_id   TEXT NOT NULL,   -- see below on what this is for a plain node    kind      TEXT NOT NULL,   -- 'refund', 'email', ...    attempts  INTEGER NOT NULL DEFAULT 0,   -- how many times the guard was asked    fired_at  TEXT,                         -- null until the effect actually fires    PRIMARY KEY (thread_id, call_id, kind));

import sqlite3CLAIM = """INSERT INTO effects_log (thread_id, call_id, kind, attempts, fired_at)     VALUES (?, ?, ?, 1, datetime('now'))ON CONFLICT (thread_id, call_id, kind) DO UPDATE        SET attempts = attempts + 1  RETURNING attempts"""def claim_effect(db, thread_id: str, call_id: str, kind: str) -> tuple[bool, int]:    """Fire the effect at most once, and count every ask.    Returns (may_fire, attempts). may_fire is True only on the first ask.    attempts keeps climbing after that - it is the record of re-execution    that the checkpoint does not keep.    Never key this on the tool arguments - Part 4's rule. Arguments cannot    distinguish a replay from a genuinely new identical request.    """    con = sqlite3.connect(db)    attempts = con.execute(CLAIM, (thread_id, call_id, kind)).fetchone()[0]    con.commit()    con.close()    return attempts == 1, attempts

I ran the flagship node from the top of this article against that, once without the ledger and once with it.

A - NO ledger (what the article's opening transcript does)  real charges fired      : 3  output done             : True   approvals: ['yes', 'yes']  checkpoints on thread   : 3  effects_log rows        : 0B - WITH the ledger  real charges fired      : 1  output done             : True   approvals: ['yes', 'yes']  checkpoints on thread   : 3  effects_log rows        : 1    thread=t-1 call=refund/req-8812 attempts=3 fired=True

Both runs return the same answer and leave the same three checkpoints. The guarded one charged the customer once instead of three times, which is Part 4's contribution and not this part's.

The new thing is the attempts=3. Three sections ago I showed that the number of body executions is not recoverable from get_state_history(), not from checkpoint metadata, and not from the raw writes table. One column in a table the runtime does not own just recovered it. That is the whole argument of this article compressed into a single integer: the information was never unknowable, it was only unrecorded by the layer everyone points their assertions at.

Three things about that table, because a careless reading of it will hurt you.

It is Part 4's design narrowed for this part, not carried over unchanged. Part 4's grant_ledger is two-phase: mint_grant() inserts, then consume() does a compare-and-swap with UPDATE ... WHERE consumed_at IS NULL. The version above collapses both phases into one upsert. That is fine for counting effects in a test and it is not a drop-in replacement for Part 4's ledger.

The attempts column is the whole fix, and I got this wrong twice before measuring it. My first version keyed rows on (thread_id, call_id, kind) and asserted on count(*). That cannot fail: the primary key pins the table at one row per logical request, so the count is 1 whether the body ran once or four times. It was an assertion that reads as evidence and checks nothing, which is the exact failure this article is about, written into the article's own remedy. My second version tried to rescue it with ON CONFLICT DO NOTHING, which quietly broke Part 4's rule that a recorded failure must clear the consumption marker so a legitimate retry can re-consume. With DO NOTHING the row survives, the retry conflicts, and the customer is stranded permanently.

The version above counts on conflict instead of ignoring the conflict. For Part 4's retry path, clear fired_at and reset attempts to 0 when you record a failure, so the next ask fires again. On Postgres the statement is the same shape with now() and TIMESTAMPTZ; the RETURNING attempts clause works on both.

This is consume-before-act, so it gives you at-most-once, not exactly-once. Part 4 chose that direction deliberately, by the rule of preferring the error that is detectable and reversible, and noted that it flips for an outbound charge. Nothing here changes that.

The snippet is SQLite, because that is what everything else in this article runs on. On Postgres use %s placeholders, TIMESTAMPTZ and now(); RETURNING works on both. Either way the helper opens and commits its own connection, which keeps it from committing whatever else a caller had pending, and costs a connection per ask. Use a pooled or context-managed connection in anything real.

call_id is the hard part, and the flagship example does not have one. The review node in every transcript in this article is a plain graph node with interrupt() calls and no tool call, so there is no tool_call_id to key on. Part 6 measured that checkpoint_ns is node:task_id, that the task id re-derives on a retry, and that it changes across a fork. So a checkpoint_ns-derived key makes the retry case idempotent and charges the fork case again. For a plain node with no tool call and no natural business key, I do not have a clean answer, and I would rather say that than invent one. If the effect matters, put it behind a tool call so it has an id, or carry an application-level idempotency key into the node from the caller.

Half two: the test creates a second entry and asserts on that table. This is the part that does not exist in any harness I surveyed.

from dataclasses import dataclass, fieldfrom langgraph.types import Command@dataclassclass SecondEntryCase:    thread_id: str    inputs: dict    approvals: list[str] = field(default_factory=list)   # one per gate    kind: str = "refund"    expected_attempts: int = 1        # how many times the guard should be asked    entries: int = 2                  # how many times to enter the threaddef _drain_gates(app, cfg, approvals: list[str], thread_id: str) -> None:    """Resume every pending gate.    Two rules here, both learned the hard way and both in earlier parts.    Loop on .interrupts, never .next: from the first resume onward .next is    empty on a thread still waiting for another approval. And resume by    interrupt IDENTITY, not position - Part 4 measured that a scalar    Command(resume=v) matches strictly by index, so reordering two    interrupt() calls between deploys resumes parked threads with the    answers swapped.    """    pending = list(approvals)    while ints := app.get_state(cfg).interrupts:        if not pending:            raise AssertionError(                f"{thread_id}: gates still pending after "                f"{len(approvals)} approvals were supplied"            )        app.invoke(Command(resume={ints[0].id: pending.pop(0)}), config=cfg)    if pending:        raise AssertionError(            f"{thread_id}: {len(pending)} unused approvals - the node has "            f"fewer gates than the case assumes"        )def assert_second_entry_safe(build_graph, saver, conn, case: SecondEntryCase) -> None:    """Run one case the way production runs it: a real checkpointer, one    thread_id, MORE THAN ONE entry, every gate resumed. Assert on the    external log, never on the returned answer.    """    app = build_graph().compile(checkpointer=saver)    cfg = {"configurable": {"thread_id": case.thread_id}}    for _ in range(case.entries):        app.invoke(case.inputs, config=cfg)        _drain_gates(app, cfg, case.approvals, case.thread_id)    attempts = conn.execute(        "SELECT coalesce(sum(attempts), 0) FROM effects_log "        "WHERE thread_id = ? AND kind = ?",        (case.thread_id, case.kind),    ).fetchone()[0]    assert attempts == case.expected_attempts, (        f"{case.thread_id}: {case.kind} guard was asked {attempts}x, expected "        f"{case.expected_attempts}. The returned answer is correct and the "        f"customer was charged once; the node still ran {attempts} times."    )

Note what the assertion reads. Not a row count, which the primary key pins at one per logical request no matter how many times the body runs. Not the returned answer, which is correct in every transcript in this article. It reads attempts, and attempts is the only number anywhere in the system that distinguishes a healthy run from the one at the top of this page.

Four details in that are corrections to versions I wrote first, and each one is a trap.

The loop runs case.entries times. My first version invoked once and then drained gates, which resumes through the first entry and never makes a second one. For a case with no gates at all, that version performed a single cold run under a name promising the opposite. Creating the second entry is the entire point, so it has to be in the control flow rather than implied by the name.

The resume uses the map form Command(resume={ints[0].id: value}). Part 4 measured that the scalar form matches interrupts strictly by position, and told readers to use the map form for any node with more than one interrupt(). A helper built to test multi-gate nodes using positional matching would introduce the exact defect class it exists to catch.

The count is scoped by kind, not just thread_id. A thread that legitimately handles two different effects should not fail an assertion about refunds.

And it raises on leftover approvals as well as on leftover gates. A case that supplies three answers to a two-gate node is a wrong test, and a wrong test that silently passes is worse than no test.

I did not want to ship the payoff pattern unexecuted in an article about assertions that pass without measuring anything, so I ran the resume logic. The specific doubt was real: the measurement above shows the resume channel holds the current value rather than accumulating one row per resume, which raises the question of whether resuming gate two loses gate one's answer.

=== n_gates=1 (MAP form, one gate per resume) ===  drain verdict     : OK after 1 resumes  approvals in state: ['yes0']          body executions: 2  ANSWERS PRESERVED IN ORDER: YES=== n_gates=2 (MAP form, one gate per resume) ===  drain verdict     : OK after 2 resumes  approvals in state: ['yes0', 'yes1']  body executions: 3  ANSWERS PRESERVED IN ORDER: YES=== n_gates=3 (MAP form, one gate per resume) ===  drain verdict     : OK after 3 resumes  approvals in state: ['yes0','yes1','yes2']  body executions: 4  ANSWERS PRESERVED IN ORDER: YES

The map form survives sequential single-gate resumes, answers stay in order, and the body still runs interrupts + 1 times, so the defect is present and the helper handles it. Run the flagship node through the ledger and attempts comes back 3 against an expectation of 1, which is the measured transcript two sections above rather than a prediction.

One precondition decides what number to expect, and it is the call_id question again. attempts counts asks per (thread_id, call_id, kind), so what you expect depends on what call_id identifies. If it is an application-level idempotency key carried in through case.inputs, it is stable across entries and expected_attempts is the number of body executions you consider correct for the whole case. If instead you key on a tool_call_id, note what Part 4 measured: that id is stable across serde and across the checkpointer, and new only when the model runs again and re-decides. Entry two is a re-decision, so you get a new id and a separate row, and the right assertion becomes one row per entry with attempts of 1 on each. Both are defensible. Picking one without noticing you picked it is how you end up with a green suite again.

I got every part of that pattern wrong before I measured it.

Without a real checkpointer, get_state raises ValueError: No checkpointer set. The entire defect class needs persistence to exist at all.

The driver loops on snapshot.interrupts, not snapshot.next. My own first probe used .next and produced a confidently wrong result table claiming the re-execution rule did not hold.

Asserting on an external log was not a design preference. The measurements forced it, because the checkpoint does not record body executions.

What to write into your eval suite tomorrow

The general form is smaller than the code above suggests. For any case in your suite that touches an irreversible effect, a budget, a counter, or a grant:

Run it twice on the same thread_id, not once on a fresh one. If your case has approval gates, resume through all of them rather than stopping at the first. Then assert an invariant on a table your checkpointer does not own.

That is the whole discipline. It does not need a new framework. It needs the harness to stop assuming the run under test is the first entry.

The four ways production re-enters a thread, and what each one costs

The eval harness exercises exactly one of these. Production uses all four, and three of them re-run node bodies.

flowchart LR
    subgraph EVAL["What the eval harness runs"]
      E1["First entry<br/>cold thread, or no<br/>checkpointer at all"]
    end

    subgraph PROD["What production runs"]
      P1["Ordinary second turn<br/>no crash, no retry"]
      P2["Resume after interrupt<br/>body re-runs per gate"]
      P3["Fork via update_state()<br/>rebuilds from an<br/>earlier checkpoint"]
      P4["Retry after crash<br/>the classical case"]
    end

    E1 --> OK["Allowance fresh<br/>nothing to restore<br/>defect cannot exist"]
    P1 --> RE["Allowance re-granted<br/>transcript retained<br/>grant re-armed"]
    P2 --> RE
    P3 --> RE
    P4 --> RE
    RE --> BLIND["Answer still correct<br/>checkpoint records nothing<br/>only an external log disagrees"]

    style E1 fill:#98D8C8,color:#2C2C2A
    style P1 fill:#FFD93D,color:#2C2C2A
    style P2 fill:#FFD93D,color:#2C2C2A
    style P3 fill:#FFD93D,color:#2C2C2A
    style P4 fill:#FFA07A,color:#2C2C2A
    style OK fill:#6BCF7F,color:#2C2C2A
    style RE fill:#E74C3C,color:#FFFFFF
    style BLIND fill:#C2185B,color:#FFFFFF

Only the last path, retry after a crash, is the one the crash-recovery and crash-consistency literature is built around. It is also the only one of the four that existing fault-injection tooling could reach. The three above it need no fault at all, which is why they arrive in production untested by anything.

The asymmetry: one durable-execution system ships a way to test its durability

This is the part of the argument I find hardest to explain away, and I looked for a reason to.

Temporal's entire safe-deployment story is replay testing. You pull recorded workflow histories from production and run them against your current workflow code in continuous integration. If the new code takes a different path than the history records, the replay fails and the build breaks. In the Python SDK the class is Replayer, with replay_workflow() and replay_workflows(). In Java it is WorkflowReplayer, and in Go it is worker.WorkflowReplayer. The following four lines are the recipe as it appears in Temporal's own documentation, not my reconstruction of it:

workflows = client.list_workflows(    f"TaskQueue={task_queue} and StartTime > '{start_time}'", limit=100)histories = workflows.map_histories()replayer = Replayer(workflows=my_workflows)await replayer.replay_workflows(histories)

A hundred recent real histories, replayed against the code you are about to deploy, on every build.

I have to be careful about what that does and does not answer, because the loose version of this comparison is unfair. Replayer detects non-determinism across code versions. Every measurement in this article uses one version of the code entered twice, and Part 4 was explicit that a Self-Restoring Bound needs neither divergent replay nor model non-determinism. A Temporal workflow carrying the Part 4 defect replays perfectly cleanly and the build stays green.

So the asymmetry is not that Temporal solved my problem. It is that Temporal built the thing that consumes durable history as a test input, and everyone else shipped the history without the harness. Once that machinery exists, pointing new assertions at it is an afternoon's work. Without it there is nowhere to put the assertion at all.

I checked the other durable-execution systems for an equivalent.

Of the six systems I could verify, exactly one ships a way to test the durability it sells.

LangGraph has every input that tool would need. It has checkpoints. It has get_state_history. It has forking through update_state() on a prior checkpoint's config. It has time travel by checkpoint_id. The histories exist, are queryable, and are already on disk. What does not exist is anything that consumes them as a test input.

The nearest thing I found is sixty-north/langchain-replay, Apache-2.0 licensed, pre-1.0, with one star on GitHub. It records model decisions, meaning which tool to call with what arguments and what text to return, and on replay yields those recorded decisions while actually executing the tools. That is a useful alternative to cassette-style HTTP recording for cost and determinism. It does not touch checkpoints and does not validate current graph code against a recorded history.

I want to phrase the claim carefully, because the loose version is false. It is not that nobody has ever replayed a LangGraph checkpoint. Time travel is a documented runtime feature and people use it to debug. It is that no tool consumes checkpoint history as a test input. The official testing page covers three patterns only, which are basic agent execution, individual node testing, and partial execution. Forking is not mentioned on it at all. Neither is crash simulation, nor durability modes, nor any assertion on a checkpoint.

Why test isolation hides second-entry defects

There is a straightforward reason nobody stumbled into this by accident, and it is not negligence. It is that the correct testing discipline, stated correctly by people who know what they are doing, removes the only condition under which the defect can exist.

Anthropic's guidance on evaluating agents, published 2026-01-09, puts it plainly. Each trial should be isolated by starting from a clean environment. Unnecessary shared state between runs causes correlated failures, and it lists leftover files, cached data and resource exhaustion as the examples.

It goes further with a concrete example. Shared state can artificially inflate performance. In some internal evaluations they observed Claude gaining an unfair advantage on some tasks by examining the git history from previous trials.

Everything in that is correct. Leftover state causes correlated failures. It inflates scores. Anyone who has debugged a flaky suite knows the shape.

The classical literature agrees and has vocabulary for it. Flaky-test research names shared persisted state as pollution, and classifies tests as polluters, victims, state-setters and brittle. Gyori, Shi, Hariri and Marinov defined a brittle test at ISSTA 2015 as one that fails in isolation and passes only after some state-setter has run.

A second-entry defect test is, by construction, a brittle test. It fails in isolation, meaning it detects nothing when the thread is cold, and only becomes meaningful after a prior entry has set up the state. Under standard test hygiene, that is a defect in the test suite to be removed.

And LangChain's own testing page prescribes the same thing for LangGraph specifically. It says that because many LangGraph agents depend on state, a useful pattern is to create your graph before each test where you use it, then compile it within tests with a new checkpointer instance. A new checkpointer per test. Every test is a first entry.

Here is the inversion I want to leave you with. In a stateless system, persisted state between runs is contamination. In a durable agent runtime, persisted state between runs is the product. The discipline that keeps a stateless test suite honest is the same discipline that makes a durable runtime's most expensive defects untestable. Both statements are true at once, which is why this gap is nobody's mistake and everybody's problem.

I am not arguing for abandoning isolation. Your correctness suite should stay isolated, and Anthropic is right about why. I am arguing that isolation is insufficient rather than wrong. You need a second suite, deliberately polluted, where the state carried between entries is the object under test rather than the thing being scrubbed.

Production reports of second-entry defects in LangGraph

I have to declare a hole here, and I would rather declare it than paper over it with a weak citation.

I found no company post-mortem or incident writeup in which an eval suite passed and the failure lived in persisted agent state. Everything below is GitHub issues plus one red-team paper. That absence is itself consistent with the thesis, since a defect nobody's tooling can see is a defect nobody writes a post-mortem about, but I am not going to present an absence as evidence.

What I did verify by opening each one:

langgraph #6208, opened 2025-09-26 by a LangGraph maintainer, still open. A node with two interrupts reruns after only one resume. The root cause is that the runtime cannot count pending resumes per task without storing interrupt identifiers in metadata, which is not supported. The maintainer's own workaround is to avoid multiple interrupts per node and chain multiple nodes instead. This is the transcript at the top of this article, and it is what ACRFence cites as cross-framework evidence.

langgraph #7417, opened 2026-04-05, open with no maintainer response. Long tool calls of roughly 180 seconds or more are silently re-executed from checkpoint on LangGraph Cloud. The reporter's diagnosis is that BG_JOB_HEARTBEAT is hardcoded at 120 in config/__init__.py and is not environment-configurable, that a queue sweep every 240 seconds marks runs stale, and that the run is reverted to pending and re-dispatched from the last checkpoint while the original is still running. They report 2 to 3 times redundant work and cost on 5 to 10 minute tool calls. Part 4 flagged that this analysis is the reporter's rather than a maintainer's, and that caveat still stands. Treat it as a credible report, not a confirmed root cause. It is still a second-entry defect with a dollar figure and correct-looking final output.

langgraph #7361, opened 2026-03-31, open. Resuming from a specific checkpoint_id becomes a full replay: the second run still runs from the beginning of the graph rather than the interrupt point. A regression in the 1.1.x line.

langgraph #4987, opened 2025-06-07, open. Fork identity, discussed with my own transcript in the fork section above. Two independent reports of the same rough area, more than a year apart, both still open.

NousResearch/hermes-agent #5563, opened 2026-04-06, closed. Over a twelve-hour session, roughly 2.6 million tokens, about 69 percent of total consumption, went to context replay overhead. In one conversation about 1.9 million tokens were consumed where roughly 190 thousand were necessary, which the reporter puts at an 89 percent waste rate. The mechanism is that when a session ends, the next session replays the full message history to maintain context. The same report includes SQLite corruption and 18 permanently lost sessions. That is Part 5's Retention Inversion with a measured production cost attached.

One more piece of evidence that cuts close and deserves its own paragraph. Catching One in Five (Zhang, Wang and Lei, arXiv:2606.10315, 2026-06) measured how well an LLM-as-judge gate detects defects in production multi-turn transaction agents. The judge caught 2 of 9 defect patterns, 22 percent. In one batch it flagged zero of 100 rounds where humans confirmed 23 distinct defects. And 113 of 114 rounds whose raw judge notes describe a confirm-gate or cart-state defect were scored as "brand voice" instead. The paper's own diagnosis is that the failure is routing rather than perception, because the gate had only three coarse axes.

Their key sentence is the one to take: judge recall falls as the required context widens from a single turn to the full session arc. That independently establishes that judges miss cross-turn state defects. Their state lives in the conversation and the shopping cart, which is dimension two rather than the runtime checkpoint, so it is partial pre-emption rather than the same finding. It is still the closest published measurement to what this article argues.

Checklist: testing a durable agent runtime

Use this in a code review of an evaluation suite, not as a reading exercise.

Does your harness create the condition at all

• At least one suite compiles the graph with a real checkpointer. If every graph in your evals is workflow.compile(), get_state raises ValueError: No checkpointer set and dimension three is unreachable.
• At least one suite reuses a thread_id across entries. A fresh thread per case is a first entry, every time.
• Cases with approval gates resume through all gates, not just the first.
• Your resume driver loops on snapshot.interrupts, never on snapshot.next. .next is empty from the first resume onward while interrupts remain pending.

Does it assert on something that can actually see the defect

• Irreversible effects are guarded by a table outside the rollback surface, and that table records how many times the guard was asked, not just whether the effect fired. A row count cannot fail; an attempt count can.
• Do not build assertions on snapshot.metadata["writes"]. It is None on every checkpoint at this pin.
• Do not try to recover a node's execution count from the thread. It is not recorded. A one-execution thread and a four-execution thread are identical on every persisted artifact.
• For every budget, counter or grant, assert its value at the start of entry two, not at the end of entry one.

Does it match how production actually re-enters

• A case exists for the ordinary second turn, with no crash. This is the trigger crash-injection tooling will never produce.
• A case exists for resume after interrupt, and a separate one for fork through update_state(). They are different code paths and the fork path hides more: it rebuilds state from a pre-approval checkpoint, so the final values look like a clean single run.
• A second-entry test that asserts first["answer"] == second["answer"] is not a second-entry test. Creating the entry is necessary and not sufficient.
• Nodes with side effects above the first interrupt() are flagged in review. The body runs interrupts + 1 times.
• Effects are idempotent per grant, keyed on something stable across a replay and distinct across a genuine new request. Part 4's rule of never keying on arguments still applies.

Keep both suites

• Your isolated correctness suite stays isolated. Anthropic's reasoning about correlated failures and inflated scores is right.
• A separate, deliberately warm suite exists where carried state is the object under test. Do not merge them, and do not let a hygiene lint delete the warm one.

What to do about the First-Entry Assumption

Three findings across Parts 4, 5 and 6 live in the runtime's own bookkeeping and need a second entry to exist: a re-granted allowance, a recoverable transcript, and a Halfway counter. This part is about why your tests will never see any of them.

The defect class is not new. Crash recovery bugs and crash-consistency bugs have been named, studied and tooled for years, and ACRFence brought the adversarial version into the agent world this March. What is new is the combination: no crash required for the sharpest of them, the runtime's bookkeeping rather than the application's data, and an evaluation discipline that is right about isolation for every reason except this one. The Halfway counter is the exception on the first clause - Part 4 measured it surviving a clean resume and rolling back on a crash, so that one does need the crash.

That is the First-Entry Assumption, and it is worth naming precisely because it is invisible from inside a passing suite. A green board means every case you wrote produced the answer you expected on its first entry into a thread that had never been entered before. In production, that is the least common way your agent runs.

Part 8 closes the series, and there is a debt to settle in it. Part 6 left its cross-thread memory section explicitly unmeasured in-article, and Part 5 promised a store-collision experiment that neither part ran. Both are still open, and after four consecutive parts where a pre-check found the ground already occupied, I am not going to assume they are still mine to claim.

References

• Gao, Y., Dou, W., Qin, F., et al. (2018). An empirical study on crash recovery bugs in large-scale distributed systems. ESEC/FSE 2018. ACM DL
• Zheng, Yang, Zhang & Quinn (2026). ACRFence: Preventing Semantic Rollback Attacks in Agent Checkpoint-Restore. arXiv:2603.20625. arXiv
• Mazaheri, P. & Mazaheri, K. (2026). AgentAtlas. arXiv:2605.20530. arXiv
• Anand, Wang, Jiang, Masson, Zheng & Zhou (2026). AEVAL. arXiv:2607.16345. arXiv
• Flynt, J. (2026). GroundEval. arXiv:2606.22737. arXiv
• Zhang, Wang & Lei (2026). Catching One in Five: LLM-as-Judge Blind Spots in Production Multi-Turn Transaction Agents. arXiv:2606.10315. arXiv
• Mishra, A. & Sharad, K. (2026). Observability for Delegated Execution in Agentic AI Systems. arXiv:2606.09692. arXiv
• Zhai, Li & Wang (2026). Revisable by Design: A Theory of Streaming LLM Agent Execution. arXiv:2604.23283. arXiv
• Grottke, M. & Trivedi, K. S. (2007). Fighting Bugs: Remove, Retry, Replicate, and Rejuvenate. IEEE Computer, 40(2), 107-109.
• Gyori, A., Shi, A., Hariri, F. & Marinov, D. (2015). Reliable testing: detecting state-polluting tests to prevent test dependency. ISSTA 2015. ACM DL
• Mohan, J., Martinez, A., Ponnapalli, S., Raju, P. & Chidambaram, V. (2018). Finding crash-consistency bugs with bounded black-box crash testing (CrashMonkey). arXiv:1810.02904. arXiv
• Fu, X., Kim, W.-H., Shreepathi, A. P., Ismail, M., Wadkar, S., Min, D. & Lee, D. (2020). WITCHER: Detecting Crash Consistency Bugs in Non-volatile Memory Programs. arXiv:2012.06086. arXiv
• LangChain. Use time travel, forking with update_state(). LangGraph docs
• Anthropic (2026-01-09). Demystifying evals for AI agents. Anthropic Engineering
• Temporal. Testing and replay, Python SDK Replayer. Temporal docs
• LangChain. Evaluate a LangGraph application. LangSmith docs
• LangChain. Testing LangGraph agents. LangGraph docs
• LangChain. Durability modes. LangGraph reference
• UK AI Security Institute. Inspect AI: checkpointing. Inspect docs
• LangChain. agentevals. GitHub
• sixty-north. langchain-replay. GitHub
• langgraph issue #6208, Do not re-execute a node that interrupted unless all of its interrupts have been resumed. GitHub
• langgraph issue #7417, Long tool calls silently re-executed from checkpoint. GitHub
• langgraph issue #7361, When resume from a specific checkpoint_id, it becomes replay. GitHub
• langgraph issue #4987, time travel then invoke, checkpoint id no longer updates. GitHub
• NousResearch/hermes-agent issue #5563, Memory persistence, token waste from session replay, state.db corruption. GitHub

All measurements in this article were run on langgraph 1.2.9, langchain 1.3.14, langchain-core 1.5.1 and langgraph-checkpoint-sqlite 3.1.0, Python 3.13.9, on 2026-08-01. No model calls and no network access were involved in any of them.

Related Articles

Agentic AI

• LangGraph Checkpoints Restore Your Limits, Not Just Your State
• LangGraph or a While Loop? You Already Have a Runtime
• State Architecture for Agent Networks: The Resume Is the Dangerous Part
• Why Your Agentic RAG System Costs 10x More Than It Should