← Back to Guides

Companion to the book Harness Engineering for Production AI Systems · this part maps to Book Ch 12

GuideFor: AI Engineers, ML Engineers, Platform Engineers, AI Systems Architects

LangGraph Interrupt Resume: Approvals Run on Stale Facts

A human approval is a checkpoint resume. It restores what the agent read before the pause, and the world kept moving while the reviewer was away.

#harness-engineering#langgraph#checkpointing#human-in-the-loop#llm-agents#production-ai-systems

On Friday the billing agent proposed a 120.00 refund for inv_9107.L3, the single sign-on (SSO) add-on that acct_4417 was charged twice, and paused at a LangGraph interrupt() for a reviewer who was on leave. On Monday the customer phoned, and a support agent refunded 120.00 from the billing console. On Tuesday the reviewer came back, opened the queue, read a proposal that was still correct as written, and clicked approve. Another 120.00 went out. Both refunds cleared at the payment provider, because 240.00 is well inside a 1,200.00 charge. This is time-of-check to time-of-use (TOCTOU), and the checkpoint is what made the window four days wide.

Nothing crashed, and nothing was retried. LangGraph resumed the agent's graph from its checkpoint exactly as designed, and that checkpoint held the ledger as it looked on Friday. I constructed this scenario from the billing agent this series has followed since Part 1, but the mechanism is real, and the code below reproduces it.

My claim is that an approval is a checkpoint resume, and a resume restores observations as faithfully as it restores commitments. So sort every field of run state by what it is, and give each kind one verb: restore, re-read, drop. Restore what the run committed to. Re-read what it observed. Drop what it concluded from the old observation. An approved proposal is the one conclusion that survives, because a person turned it into a commitment, and it survives only together with the evidence it was approved on. Then prove each rule runs on the ways back into the effect, the human one included, and say which ways you tested. A restore rule that only a startup check reads is a comment. Revalidating a stale approval is already published, and I credit that work below. What this part adds is the sort that chooses each field's verb, the approval following from it, a graph walk and a resume test that check the rules ran, and the finding that my own book's rules never did.

Why a LangGraph interrupt() approval is a checkpoint resume

Most people picture human-in-the-loop (HITL) approval as a pause button: the run stops, a person decides, and the run continues where it was. LangChain's middleware states it that way: on approve it will "Execute the tool with the original arguments as proposed by the agent." The OpenAI Agents SDK says "The resumed run continues where it left off". This blog said the same thing about crash recovery in April: "the next run resumes exactly where it left off - no duplicated side effects, no lost work, no contradictory outputs from partial runs" (State Management for Agentic Systems).

LangGraph's own documentation files the two cases together. Its checkpointer guide has a table, "Choose how to continue a thread", with these rows next to each other:

  • Continue after an error: "Loads the latest checkpoint and retries unfinished work."
  • Answer an interrupt(): "Supplies the interrupt's return value. The interrupted node starts again from its beginning."

A crash recovery and an approval go through one mechanism: both load the latest checkpoint and restart the work it left unfinished, which at an interrupt is one node. Neither re-runs the nodes upstream of it, so neither reads the ledger again. The LangChain and OpenAI pages say nothing about reading the world again before the approved call runs. That April promise mostly holds for a crash a few seconds into a run, because little outside moves in a few seconds. It breaks for a pause that lasts as long as a person takes to answer, which can be days. Both entries below land on the same saved state.

d2
direction: down
classes: {
  entry: {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
  saved: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  stale: {style: {fill: "#FFD93D"; stroke: "#C9A400"; font-color: "#2C2C2A"}}
  act: {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
}
crash: "Worker died at 02:14\noperator restarts it" {class: entry}
human: "Reviewer clicks approve\nfour days later" {class: entry}
ckpt: "Latest checkpoint\nfacts read on Friday\nproposal made on Friday" {class: saved}
node: "One node restarts\nupstream nodes do not run" {class: stale}
effect: "issue_refund\non Friday's ledger" {class: act}
crash -> ckpt: "load"
human -> ckpt: "load"
ckpt -> node
node -> effect

Both arrows end at the same box. Whatever the checkpoint holds about the world is what the refund acts on, whichever of the two brought the run back.

Wrong way: resume the interrupt on checkpointed facts

Here is the smallest version of the agent that shows it. Billing is a plain class with one rule the provider enforces: a refund may not exceed what is left on the charge. Refund amounts come from the ledger, never from the model, because Part 10 narrowed issue_refund to take a line id.

python
from __future__ import annotationsimport hashlibimport jsonimport sqlite3import tempfilefrom decimal import Decimalfrom pathlib import Pathfrom typing import Any, TypedDict, get_type_hintsfrom langgraph.checkpoint.sqlite import SqliteSaverfrom langgraph.graph import END, START, StateGraphfrom langgraph.types import Command, interruptclass ProviderRefused(Exception):    passclass Billing:    """The ledger and the payment provider. The provider refuses only what it    models: a refund larger than what is left on the charge."""    def __init__(self) -> None:        self.lines = {            "inv_9107.L1": ("acct_4417", "ch_9107", "team plan", Decimal("960.00")),            "inv_9107.L2": ("acct_4417", "ch_9107", "sso add-on", Decimal("120.00")),            "inv_9107.L3": ("acct_4417", "ch_9107", "sso add-on", Decimal("120.00")),        }        self.charges = {"ch_9107": Decimal("1200.00")}        self.refunds: list[tuple[str, Decimal, str]] = []        self.contact = {"acct_4417": "billing@example.com"}        self.reads = 0    def refunded(self, charge_id: str) -> Decimal:        return sum((a for c, a, _ in self.refunds if c == charge_id), Decimal("0.00"))    def read_facts(self, line_id: str) -> dict[str, Any]:        self.reads += 1        account, charge, item, amount = self.lines[line_id]        return {            "line": {"id": line_id, "item": item, "amount": str(amount)},            "charge": charge,            "charge_refunded": str(self.refunded(charge)),            "contact": self.contact[account],        }    def refund(self, line_id: str, by: str) -> None:        _, charge, _, amount = self.lines[line_id]      # amount from the ledger        if self.refunded(charge) + amount > self.charges[charge]:            raise ProviderRefused(f"{charge}: refund exceeds what is left")        self.refunds.append((charge, amount, by))def show_refunds(billing: Billing) -> str:    return ", ".join(f"{a} ({by})" for _, a, by in billing.refunds) or "none"

This graph reads the facts, proposes, pauses, and refunds. DECLARED is a restore spec of the kind the book's chapter writes, and undeclared is the kind of startup check that guards it: every field of run state must carry a rule.

python
class Run(TypedDict, total=False):    run_id: str    line_id: str    facts: dict[str, Any]    proposal: dict[str, Any]    approved: boolDECLARED = {    "run_id": "restored",    "line_id": "restored",    "facts": "rederived",     # the ledger as it is now    "proposal": "discarded",  # made against facts that have since moved    "approved": "restored",}def undeclared(spec: dict[str, str], state: type) -> list[str]:    return sorted(set(get_type_hints(state)) - set(spec))def build_naive(billing: Billing, saver: SqliteSaver) -> Any:    def read(s: Run) -> Run:        return {"facts": billing.read_facts(s["line_id"])}    def propose(s: Run) -> Run:                         # the model, stood in        return {"proposal": {"tool": "issue_refund", "line_id": s["line_id"]}}    def approve(s: Run) -> Run:        return {"approved": interrupt(s["proposal"]) == "approve"}    def refund(s: Run) -> Run:        if s["approved"]:            billing.refund(s["proposal"]["line_id"], by="agent")        return {}    g = StateGraph(Run)    for name, fn in [("read", read), ("propose", propose),                     ("approve", approve), ("refund", refund)]:        g.add_node(name, fn)    g.add_edge(START, "read")    g.add_edge("read", "propose")    g.add_edge("propose", "approve")    g.add_edge("approve", "refund")    g.add_edge("refund", END)    return g.compile(checkpointer=saver)

In the run below, the graph pauses, the console refund lands, and a second process picks the thread up from the same SQLite file and answers the interrupt. Every output block in this part is real output from LangGraph 1.2.22 with langgraph-checkpoint 4.3.2 and langgraph-checkpoint-sqlite 3.1.3.

python
def saver_at(path: Path) -> SqliteSaver:    return SqliteSaver(sqlite3.connect(str(path), check_same_thread=False))def naive_run() -> None:    billing = Billing()    store = Path(tempfile.mkdtemp()) / "threads.db"    config = {"configurable": {"thread_id": "T-4471"}}    print("undeclared fields:", undeclared(DECLARED, Run))    app = build_naive(billing, saver_at(store))    app.invoke({"run_id": "T-4471", "line_id": "inv_9107.L3"}, config)    print("paused at:", app.get_state(config).next, " billing reads:", billing.reads)    billing.refunds.append(("ch_9107", Decimal("120.00"), "console"))    print("during the pause, the console refunds 120.00 on ch_9107")    later = build_naive(billing, saver_at(store))     # a new process, days later    later.invoke(Command(resume="approve"), config)    print("refunds on ch_9107:", show_refunds(billing))    print("billing reads:", billing.reads)naive_run()
text
undeclared fields: []paused at: ('approve',)  billing reads: 1during the pause, the console refunds 120.00 on ch_9107refunds on ch_9107: 120.00 (console), 120.00 (agent)billing reads: 1

Every field has a rule, and the check passes. Only one ledger read happened, on Friday, and the customer got 240.00 back for a 120.00 mistake. According to the spec, facts is rederived and proposal is discarded. Nothing in the graph does either.

Here are the two timelines side by side: what the run held, and what the billing system held.

d2
grid-rows: 1
grid-gap: 30
classes: {
  run: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  world: {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
  bad: {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
}
held: "What the checkpoint holds" {
  grid-columns: 1
  label.near: top-center
  f1: "Fri: facts read\ncharge_refunded 0.00" {class: run}
  f2: "Fri: proposal\nrefund inv_9107.L3" {class: run}
  f3: "Mon: unchanged" {class: run}
  f4: "Tue: approve resumes\non Friday's facts" {class: bad}
}
ledger: "What billing holds" {
  grid-columns: 1
  label.near: top-center
  w1: "Fri: 0.00 refunded" {class: world}
  w2: "Fri: 0.00 refunded" {class: world}
  w3: "Mon: console refunds 120.00" {class: world}
  w4: "Tue: agent refunds 120.00\n240.00 refunded" {class: bad}
}

Nothing in the left column changes after Friday. That is the checkpoint doing its job, and it is the whole failure.

Part 10 left one check owed at act time: whether the line was already refunded. That check, and the conditional write in the race section below, protect an invariant the code knows to test, and the demo leaves them out on purpose so the mechanism stays visible. They are not enough on their own. They refuse; they cannot ask the reviewer again with the new facts. And they only see what the code thought to check. A reviewer's yes can rest on evidence the ledger has no invariant for, such as an open dispute on the account or a note from the phone call, and only a re-read of what the approval depended on catches that. The invariant checks protect the ledger. reconcile protects what the approval meant.

Why checkpoint restore rules never run: the book's own spec

I first saw this in my own book's code, so it belongs here in full. Chapter 12 of Harness Engineering for Production AI Systems writes the restore spec as a TOML file with a reason on every rule. facts is rederived because "A refund banded against a four-hour-old ledger is a decision made on evidence that has since changed", and proposal is discarded. Three startup checks compare the spec with the state object, the tool registry and the compiled graph.

I ran the sequence above against the companion repository at tag ch12-state-checkpoints-resume, on LangGraph 1.2.22. Chapter 12's graph paused at approve, a console refund of inv_9002 landed in the example ledger, and the resume refunded it again, with read and propose each running once:

console
paused at ('approve',) reads 1 proposals 1after resume: ledger [('inv_9002', Decimal('89.00'), 'console'),                      ('inv_9002', Decimal('89.00'), 'harness')]reads 1 proposals 1

Searching the package finds no runtime reader of the rederived list, of rule_for(), or of the DISCARDED value. One test asserts the spec's contents: budget in its restored list and facts in its rederived list, a green test with no running rule behind it. All three startup checks read names, plus the replay column and the registry's idempotency flags, and none of them reads the restore column. No node in that graph writes facts at all. The probe injected the console entry, added the 'console' and 'harness' labels, and counted calls; reads counts the read node's fetches. It shows that read and propose did not run again. It does not show that running the rules would have stopped this refund, because the example's proposer and ledger do not consult each other.

Two more details make the same point. In that code, the approval travels in the band field, which the spec declares rederived with the reason "A band read back from a checkpoint is a grant that outlived the reason it was given." And the interrupt shows the reviewer the tool name and the run id, but not the amount. Part 9 bound the approval to the bytes the reviewer saw, and here the reviewer saw almost none of them.

None of this makes the chapter's rules wrong. They are the right rules, written as declarations with no executor. That is the defect Part 3's loop-closure check reports for guardrails, a verdict that nothing reads, now showing up in a restore spec. Each of the spec's three rules is paired below with what actually runs at an approval resume.

d2
grid-rows: 1
grid-gap: 30
classes: {
  spec: {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
  ran: {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
  none: {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
}
declared: "Declared in checkpoint-spec.toml" {
  grid-columns: 1
  label.near: top-center
  d1: "budget, plan: restored" {class: spec}
  d2: "facts: rederived" {class: spec}
  d3: "band: rederived\nproposal: discarded" {class: spec}
}
executed: "What runs at an approval resume" {
  grid-columns: 1
  label.near: top-center
  e1: "restored: the default,\nplus channel writes for budget" {class: ran}
  e2: "never re-read: no node\nwrites facts" {class: none}
  e3: "read back and executed:\napprove writes band, refund reads it" {class: none}
}

Only the restored rules hold, because restoring is what LangGraph does to every channel by default. For budget it also takes code, since every counter change has to go through the state channel. The rules that ask for something different never ran.

The Restore Line: restore commitments, re-read observations, drop conclusions

The Restore Line is the book's term for a declared split of run state. For each field, it says whether a checkpoint restores it or a resumed run re-derives it, with discarding as a third option. The chapter gives a reason for each field, and its REDERIVED rule already carries the idea that a value can be true four hours ago and false now. What it does not give is a general rule for choosing a field's side. I choose by what the field is:

  • A commitment is something the run already decided or spent: its identity, the customer's request, the budget it has used, the plan Part 11 froze before untrusted text arrived. Restore it, never re-derive it. A forward resume can take it from the checkpoint. If it authorizes an effect, it also needs a record outside the checkpoint, because a rewind or a fork of the thread restores the checkpoint's copy too.
  • An observation is what the world looked like when the run read it: the ledger, the account, the invoice. Re-read it from the system of record before anything acts on it, and keep the restored copy as the record of what the run decided on. The approval's digest is computed from that restored copy.
  • A conclusion is what the run inferred from an observation: the model's proposal, a band computed from the facts. Drop it, and let the run infer it again from the fresh observation.

The sort explains a failure this blog showed in July and the one in this part, from opposite directions. In July, the post LangGraph Checkpoints Restore Your Limits, Not Just Your State named the Self-Restoring Bound: an allowance "re-derived from checkpointed state on every entry, rather than stored as a spent total". Its recursion_limit case is a commitment re-derived. The step counter comes back honestly. The stop point, which should be fixed once and carried, is recomputed from config at every entry, so the bound arrives full. Tuesday's refund is the opposite: an observation treated as a commitment, so the ledger arrives stale.

July's other case needs a second question. A grant restored unspent from a rewound checkpoint has the right verb, restore, read from the wrong store. July answered that question. There the approval is not restored at all: it is re-delivered to the run as a resume value, replayed forward, and the fix is an approval_id row outside the rollback surface, consumed once by compare-and-swap. The record that a commitment was spent lives where the rollback cannot reach it. So the full rule has two parts, which verb a field gets and which store the verb reads from, and the store half is July's. This part's fix re-reads the observation from the system of record. Neither fix trusts the checkpoint for something the checkpoint cannot know. The book's rederived covers two different things: re-reading an observation and inferring a conclusion again. The sort splits them. What it adds to the book's three verbs is the rule that chooses the verb, the approval following from that rule, and the evidence that the book's own rules never ran. Some fields hold more than one kind. The book's spec restores context on purpose. It holds retrieved text, which is an observation, but the spec's reason is that re-deriving it "means re-running retrieval, which spends budget and produces a different window". Restoring it brings back the untrusted spans too, and the chapter says that is safe only because the plan is restored, which is Part 11's Frozen Plan. LangGraph's messages mixes tool results and model output in one list. Restore fields like these as history, and never let an effect's precondition come from them; take it from a fresh read. Each kind of field gets a column of its own below.

d2
grid-rows: 1
grid-gap: 24
classes: {
  c: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  o: {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
  x: {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
  bug: {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
}
commit: "Commitment: restore" {
  grid-columns: 1
  label.near: top-center
  c1: "run_id, line_id\nbudget spent, frozen plan\nthe approval" {class: c}
  c2: "Recompute the stop point and\nthe bound arrives full (July)" {class: bug}
}
observe: "Observation: re-read" {
  grid-columns: 1
  label.near: top-center
  o1: "facts: ledger, account,\ninvoice lines" {class: o}
  o2: "Restore it and the ledger\narrives stale (this part)" {class: bug}
}
conclude: "Conclusion: drop" {
  grid-columns: 1
  label.near: top-center
  x1: "proposal, band\ncomputed from facts" {class: x}
  x2: "Keep it and an old inference\nacts after the facts moved" {class: bug}
}

Durable execution engines take the other side on observations, and on purpose. Temporal replays a workflow from its event history and recreates its state "to what it was immediately before the crash". DBOS recovers "from the last completed step". Inngest uses "saved results for completed steps without running their callbacks again". Restate persists the result of any non-deterministic operation you wrap in run. All of them restore recorded observations, because replaying the past must reach the same state every time. That is correct for reconstructing the past, and wrong for the next consequential decision after a wait. Inngest's own example loads an order, waits up to six hours for an event, and refunds the order it loaded before the wait. In those engines the fresh read has to be a new step after the wait, and the same is true in LangGraph.

Stale approvals: bind each approval to the evidence it was given on

At the approval, the sort seems to break. A proposal is a conclusion, so it should be dropped, but the approval is about that proposal. Drop it and the reviewer's yes refers to nothing. Keep it and Tuesday's refund goes out.

Ask instead what the approval actually is. A person read a proposal and some facts, and committed the run to acting on that proposal on those facts: a commitment with a condition attached. So the approval is restored, and it holds only while the facts it was given on still hold. On resume, the run re-reads them and compares. If they moved, the approval is void, the proposal is dropped, and the run proposes again from the new facts and asks again.

This part did not invent revalidating an approval. Peng and Wu's MasuGate paper defines a stale approval, "where the human decision no longer authorizes the effect unless the system preserves or revalidates the state that justified it", and uses refund history as an example of the state refund governance depends on. A community post on AWS Builder Center from September 2026 carries "the domain-state digest" in the checkpoint and refuses to resume when the shop database changed: "The decision was made about a world that no longer exists." AID-Guard revalidates provider state at commit across retry and recovery. Practitioner posts made the same point in April, re-checking the world before a side effect once time has passed, and the Loopjacking paper says plainly that this body of work rules out a novelty claim on stale authorization. What the sort adds is that the approval follows from the same rule as every other field, and that rule also explains the July bug.

Two choices decide whether this works in practice.

What the digest covers. Hash only the approval's read set: the facts the reviewer was shown and the decision depended on. Here that is the line and the amount already refunded on its charge. A digest of everything the run read voids an approval when the customer changes their email address, and reviewers soon learn to approve the second request without reading it.

What voids an approval. The June post on HITL at scale gave every pending approval an expiry by tier. An asynchronous approval waits four hours, then the run keeps its state and escalates. A synchronous one waits thirty minutes, then fails closed. Expiry bounds how long a run can sit, and it is worth keeping. If refunds sit in June's Tier 2, expiry would have escalated this ticket long before Tuesday, and it would have done the same if nothing had changed. And it would have let the double refund through if the customer had phoned twenty minutes into the pause, inside either window. Time only stands in for staleness, and the evidence is what actually changed. Compare the three bindings in the diagram, and look at which one stops the refund.

d2
grid-rows: 1
grid-gap: 24
classes: {
  q: {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
  pass: {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
  stop: {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
}
bytes: "Bound to the bytes (Part 9)" {
  grid-columns: 1
  label.near: top-center
  b1: "proposal digest on Tue\n= digest on Fri" {class: q}
  b2: "refund runs: 240.00" {class: pass}
}
clock: "Bound to a clock (June)" {
  grid-columns: 1
  label.near: top-center
  t1: "console refund at 20 min\ninside a 30-minute expiry" {class: q}
  t2: "refund runs: 240.00" {class: pass}
}
evidence: "Bound to the evidence" {
  grid-columns: 1
  label.near: top-center
  e1: "charge_refunded\n0.00 on Fri, 120.00 on Tue" {class: q}
  e2: "approval void,\nrun asks again" {class: stop}
}

Across the pause the proposal bytes never change, and a refund twenty minutes in is inside the expiry. Only the evidence binding sees that the world moved, and it sees it at any age.

Right way: a reconcile node after the interrupt

To fix this, stop writing the Restore Line as a file that code is supposed to obey, and make it the input of the code that does the restoring. RESTORE_LINE below is read by restore(), which re-reads every observation through its entry in READERS and clears every conclusion. On the approval path, nothing else decides what a resume trusts. If a reader fails, restore() raises and the refund does not run, so a broken ledger read fails closed instead of falling back to the checkpoint. READ_SET names what the approval depends on, close to what MasuGate calls the policy read set, and evidence() hashes only that. Part 9's digest binds the approval to the proposal's bytes. This one binds it to the evidence, which the bytes cannot show, and a full harness keeps both.

python
COMMITMENT, OBSERVATION, CONCLUSION = "commitment", "observation", "conclusion"RESTORE_LINE: dict[str, tuple[str, str]] = {    "run_id":   (COMMITMENT,  "the run's identity; everything else is keyed on it"),    "line_id":  (COMMITMENT,  "what the customer asked for, fixed at intake"),    "approval": (COMMITMENT,  "a person decided; it holds only with its evidence"),    "rejected": (COMMITMENT,  "a person said no; the run never asks again"),    "facts":    (OBSERVATION, "the ledger as it is now, not when it was read"),    "proposal": (CONCLUSION,  "inferred from facts that are about to be re-read"),    "voided":   (CONCLUSION,  "why the last reconcile dropped an approval"),}READERS = {"facts": lambda billing, s: billing.read_facts(s["line_id"])}READ_SET = ("line", "charge_refunded")    # what the approval depended ondef evidence(facts: dict[str, Any]) -> str:    shown = {k: facts[k] for k in READ_SET}    blob = json.dumps(shown, sort_keys=True).encode()    return hashlib.sha256(blob).hexdigest()[:12]def restore(state: dict[str, Any], billing: Billing) -> tuple[dict[str, Any], str]:    """Apply every rule. Returns the update and why an approval was voided."""    update: dict[str, Any] = {}    for name, (kind, _why) in RESTORE_LINE.items():        if kind == OBSERVATION:            update[name] = READERS[name](billing, state)    # a failed read raises        elif kind == CONCLUSION:            update[name] = None    approval, voided = state.get("approval"), ""    if approval and evidence(update["facts"]) != approval["evidence"]:        was = state["facts"]["charge_refunded"]        now = update["facts"]["charge_refunded"]        voided = f"evidence changed (charge_refunded {was} -> {now})"        update["approval"] = None    return update, voided

One node, reconcile, goes between the approval and the effect. It applies the line and routes: refund if the approval survived, propose again if it was voided, stop if the reviewer said no. Now the interrupt shows the reviewer the facts in the read set, so the digest covers what the interrupt showed, provided nobody updated the paused thread. To bind it to the screen itself, send the shown digest back in the resume value and compare it. refund stands in for Part 10's act. In the full harness a re-proposal after a void goes back through Part 9's band and gates like any other proposal, and The Frozen Plan from Part 11 is a commitment, restored and never frozen again.

python
class Resumable(TypedDict, total=False):    run_id: str    line_id: str    facts: dict[str, Any]    proposal: dict[str, Any] | None    approval: dict[str, Any] | None    rejected: bool    voided: str | Nonedef build(billing: Billing, saver: SqliteSaver) -> Any:    def read(s: Resumable) -> Resumable:        return {"facts": billing.read_facts(s["line_id"])}    def propose(s: Resumable) -> Resumable:        return {"proposal": {"tool": "issue_refund", "line_id": s["line_id"]}}    def approve(s: Resumable) -> Resumable:        """The pause and what it shows. Anything else here runs again on resume."""        shown = {"proposal": s["proposal"],                 "facts": {k: s["facts"][k] for k in READ_SET}}        answer = interrupt(shown)        if answer != "approve":            return {"approval": None, "rejected": True}        return {"approval": {"proposal": s["proposal"],                             "evidence": evidence(s["facts"]), "by": "reviewer"}}    def reconcile(s: Resumable) -> Resumable:        update, voided = restore(dict(s), billing)        return {**update, "voided": voided}    def refund(s: Resumable) -> Resumable:        billing.refund(s["approval"]["proposal"]["line_id"], by="agent")        return {}    def route(s: Resumable) -> str:        if s.get("rejected"):            return END        return "refund" if s.get("approval") else "propose"    g = StateGraph(Resumable)    for name, fn in [("read", read), ("propose", propose), ("approve", approve),                     ("reconcile", reconcile), ("refund", refund)]:        g.add_node(name, fn)    g.add_edge(START, "read")    g.add_edge("read", "propose")    g.add_edge("propose", "approve")    g.add_edge("approve", "reconcile")    g.add_conditional_edges("reconcile", route, ["refund", "propose", END])    g.add_edge("refund", END)    return g.compile(checkpointer=saver)

approve holds nothing but the pause and a deterministic digest of state it was handed, which keeps its Replay Debt at zero. Replay Debt is the book's second term for this chapter: the effects a node already performed when the run stopped, which a resume performs again. A resume performs them again because it restarts the node. LangGraph's interrupt() docstring says so: "The graph resumes from the start of the node, re-executing all logic." A notification or a counter placed in approve would run once per resume, and the July post and the June post both cover that rule in detail. A re-read at the top of approve would also run on every resume, because a resume re-runs that line, and storing the digest earlier, in propose, might make that comparison work. I have not tested that design, and I give the re-read a node of its own anyway. In approve the read also runs on the first pass, before anyone has answered, and a failing read inside the interrupt node repeats on every resume attempt. Giving the re-read a node of its own after the pause keeps the two reads apart: the one the reviewer saw, and the one the refund will act on.

In the topology below, every edge into refund comes from reconcile. A crash after reconcile can still resume at refund itself, which the race section deals with.

d2
direction: down
classes: {
  step: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  pause: {style: {fill: "#FFD93D"; stroke: "#C9A400"; font-color: "#2C2C2A"}}
  rec: {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
  act: {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
  end: {style: {fill: "#95A5A6"; stroke: "#6F7C7D"; font-color: "#2C2C2A"}}
}
read: "read: facts" {class: step}
propose: "propose: proposal" {class: step}
approve: "approve: interrupt only" {class: pause}
reconcile: "reconcile: re-read facts\ndrop proposal\ncompare evidence" {class: rec}
refund: "refund" {class: act}
stop: "end" {class: end}
read -> propose: "1"
propose -> approve: "2"
approve -> reconcile: "3. resume lands here"
reconcile -> refund: "4a. approval holds"
reconcile -> propose: "4b. void: ask again"
reconcile -> stop: "4c. rejected"
crash: "crash after reconcile" {class: end}
crash -> refund: "crash resume" {style: {stroke-dash: 4}}

That loop from reconcile back to propose is the void path. It costs one model call and one more review, and it only runs when the world moved.

Running both pauses again, once with the console refund and once with nothing changed except the customer's contact email:

python
def resumable_run(change: str) -> None:    billing = Billing()    store = Path(tempfile.mkdtemp()) / "threads.db"    config = {"configurable": {"thread_id": "T-4471"}}    build(billing, saver_at(store)).invoke(        {"run_id": "T-4471", "line_id": "inv_9107.L3"}, config)    if change == "console refund":        billing.refunds.append(("ch_9107", Decimal("120.00"), "console"))    else:        billing.contact["acct_4417"] = "ap@example.com"    later = build(billing, saver_at(store))    later.invoke(Command(resume="approve"), config)    state = later.get_state(config)    if state.values.get("voided"):        print("  approval voided:", state.values["voided"])        shown = state.tasks[0].interrupts[0].value["facts"]        print("  paused again; the reviewer now sees charge_refunded",              shown["charge_refunded"])        later.invoke(Command(resume="reject"), config)    print("  refunds on ch_9107:", show_refunds(billing))    print("  billing reads:", billing.reads)print("a console refund during the pause")resumable_run("console refund")print("only the contact email changed during the pause")resumable_run("contact email")
text
a console refund during the pause  approval voided: evidence changed (charge_refunded 0.00 -> 120.00)  paused again; the reviewer now sees charge_refunded 120.00  refunds on ch_9107: 120.00 (console)  billing reads: 3only the contact email changed during the pause  refunds on ch_9107: 120.00 (agent)  billing reads: 2

In the first run there is one refund, and it is the console's. Asked again with the new number in front of them, the reviewer said no. That no is the script's choice, not part of the mechanism. An approve here would be a new decision made on the new evidence, and the refund would run. Three ledger reads happened: before the pause, at the approval resume, and at the reject resume. Bound that loop in production: after a second void on one ticket, hand it to a person, as Part 9's hand-off does, instead of asking a third time. In the second run, the approval survived a change that the decision never depended on. One rule produced both outcomes, and the approval had no clock on it.

Testing the LangGraph approval resume path

A rule that runs only on the paths someone remembered is not much better than a rule that never runs. Chapter 12's checks compare the spec with the state object, the tool registry and the compiled graph's node names. That catches a field nobody wrote a rule for, but it cannot catch a path into the effect that skips the code applying the rules, which is the exact defect above.

So the check has two halves. At startup it walks the compiled graph from every node a resume can land on, here the one pause, following edges without passing through reconcile, and reports any effect it reaches. PAUSES and EFFECTS are declared by hand, so they get reviewed with the graph like any other declaration. Because the walk trusts the name reconcile, the second half tests behaviour. It drives the approval resume twice. With nothing changed, it fails if an observation was not read again or a conclusion survived. With the console refund landed during the pause, it fails if the refund ran, or if the run did not pause to ask again. Both halves also run the book-style spec, with rederived read as an observation and discarded as a conclusion.

python
EFFECTS = {"refund"}     # declared by hand, so review them with the graphPAUSES = {"approve"}def restore_line_problems(rules: dict[str, Any], readers: dict[str, Any],                          state: type, app: Any) -> list[str]:    found = [f"{f} has no rule" for f in undeclared(rules, state)]    found += [f"{f} is an observation with no reader" for f, (kind, _) in rules.items()              if kind == OBSERVATION and f not in readers]    edges: dict[str, set[str]] = {}    for e in app.get_graph().edges:        edges.setdefault(e.source, set()).add(e.target)    for start in sorted(PAUSES):        seen, todo = set(), [start]        while todo:                                   # walk, but never through reconcile            node = todo.pop()            for nxt in edges.get(node, ()):                if nxt == "reconcile" or nxt in seen:                    continue                seen.add(nxt)                todo.append(nxt)        for effect in sorted(EFFECTS & seen):            found.append(f"{effect} is reachable from {start} without reconcile")    return founddef approval_resume_problems(builder: Any, rules: dict[str, Any]) -> list[str]:    """Drive the human resume path twice and check what it actually did."""    config = {"configurable": {"thread_id": "test"}}    def paused(billing: Billing) -> Any:        app = builder(billing, saver_at(Path(tempfile.mkdtemp()) / "t.db"))        app.invoke({"run_id": "test", "line_id": "inv_9107.L3"}, config)        return app    billing = Billing()                         # nothing changes during the pause    app = paused(billing)    before = billing.reads    app.invoke(Command(resume="approve"), config)    values = app.get_state(config).values    found = []    for name, (kind, _) in rules.items():        if kind == OBSERVATION and billing.reads == before:            found.append(f"{name} was not re-read on the approval resume")        if kind == CONCLUSION and values.get(name):            found.append(f"{name} survived the approval resume")    billing = Billing()                         # the evidence moves during the pause    app = paused(billing)    billing.refunds.append(("ch_9107", Decimal("120.00"), "console"))    app.invoke(Command(resume="approve"), config)    if any(by == "agent" for _, _, by in billing.refunds):        found.append("refund ran after its evidence changed")    elif app.get_state(config).next != ("approve",):        found.append("the void did not ask the reviewer again")    return found# The book's "rederived" covers re-reading and re-inferring.# Here the one rederived field, facts, is a read: map it to an observation.BOOK_KINDS = {"restored": COMMITMENT, "rederived": OBSERVATION,              "discarded": CONCLUSION}book_rules = {f: (BOOK_KINDS[v], "") for f, v in DECLARED.items()}naive = build_naive(Billing(), saver_at(Path(tempfile.mkdtemp()) / "t.db"))fixed = build(Billing(), saver_at(Path(tempfile.mkdtemp()) / "t.db"))for label, rules, readers, state, app, builder in [        ("naive graph", book_rules, {}, Run, naive, build_naive),        ("fixed graph", RESTORE_LINE, READERS, Resumable, fixed, build)]:    print(label)    problems = restore_line_problems(rules, readers, state, app)    problems += approval_resume_problems(builder, rules)    for problem in problems or ["nothing"]:        print("  " + problem)
text
naive graph  facts is an observation with no reader  refund is reachable from approve without reconcile  facts was not re-read on the approval resume  proposal survived the approval resume  refund ran after its evidence changedfixed graph  nothing

Earlier the naive graph passed undeclared with an empty list, and here it fails five times. Its spec names facts as rederived and gives no code to re-read it. Its last three findings are the Tuesday refund, found in a test with no model and no reviewer: the ledger was not read again, the proposal outlived the evidence it was made on, and the refund ran after the console's. The test counts total reads, which is enough here because the graph has one observation. With several, count reads per reader. The walk stops at a different place in each graph:

d2
grid-rows: 1
grid-gap: 30
classes: {
  n: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  r: {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
  e: {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
  ok: {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
}
naive: "Naive graph" {
  grid-columns: 1
  label.near: top-center
  na: "approve" {class: n}
  nr: "refund: reached\nwithout reconcile" {class: e}
}
fixed: "Fixed graph" {
  grid-columns: 1
  label.near: top-center
  fa: "approve" {class: n}
  fr: "reconcile: the walk\nstops here" {class: r}
  fe: "refund: not reached" {class: ok}
}

To the walk, reconcile is a wall, and any effect it can still reach is an effect some resume can reach without fresh facts. Together the walk and the test cover the approval resume, with and without a change during the pause. They do not cover a rewind to an older checkpoint, which is why commitments that authorize effects need a record outside it, or a crash between reconcile and refund, which the next section deals with.

Closing the race between re-reading state and the refund

reconcile reads the ledger, LangGraph writes a checkpoint, and then refund runs. A console refund that lands between those two steps goes unseen. On the normal path that window is one superstep long. On the crash path it runs from reconcile's read to the restart, because a crash in that gap resumes at refund without passing reconcile again. Either way it is far shorter than a four-day pause, and it is still there.

Only the system of record can close it, because only it sees every writer. If you own the ledger, make the refund a conditional write that fails when charge_refunded is not what reconcile read. If a payment provider owns it, find out which invariants it actually enforces. Stripe refuses a second full refund of the same charge with charge_already_refunded. A second partial refund that still fits under the charge is allowed, since a charge can be refunded in parts, and that is the Tuesday case. An idempotency key does not help either: Stripe describes it as a key "that the server uses to recognize subsequent retries of the same request", and the console's refund was never your request. ProviderRefused in listing 1 is that kind of invariant: it stops a refund that would pass the charge total, and nothing smaller. Chapter 12's effect log has the same blind spot, since it recognises the harness's own repeated calls and nothing else. The window left between the re-read and the refund looks like this:

d2
direction: right
classes: {
  t: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  gap: {style: {fill: "#FFD93D"; stroke: "#C9A400"; font-color: "#2C2C2A"}}
  close: {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
}
r: "reconcile\nreads 0.00" {class: t}
g: "checkpoint write\nwindow: one superstep\nor until the restart" {class: gap}
f: "refund" {class: t}
c: "conditional write:\nfail if not 0.00" {class: close}
r -> g -> f -> c

Restore, re-read, drop: one rule for every resume

Three failures in this part look unrelated. Tuesday's refund acted on a ledger that was four days old. July's recursion limit arrived full after every resume. Chapter 12's restore spec passed every check and never ran. Each one gave a field the wrong verb, or wrote the right verb where no code read it. An approval needs no exception to the sort, though it needs its own check: a person turned a conclusion into a commitment, so it is restored, and it holds only while the observations it was given on still hold. Write the verb and the store for every field, put the rules in the code that restores, and test the approval resume as hard as the crash. Every page quoted at the top is right that a resume continues where it left off. For an approval, that is the defect: it continues on what the run knew when it stopped.

Checklist: resuming an agent after a human approval

  1. List every field of run state, and mark each one a commitment, an observation, or a conclusion. Write one sentence for each, saying why.
  2. Restore commitments. A forward resume can take them from the checkpoint. Anything that authorizes an effect, such as a spend total or an approval, also needs a record that a rewind or fork of the thread cannot bring back, like the July post's single-use approval row.
  3. Re-read observations from the system of record on every resume that reaches an effect, not only after a crash. An approval is a resume. If the read fails, fail closed. Keep the restored copy as the record of what was decided on, and never take an effect's precondition from it.
  4. Drop conclusions and infer them again from the fresh observations. Never execute a proposal that outlived its evidence.
  5. Store each approval with a digest of its read set: the facts the reviewer was shown and the decision depended on. Void it when the re-read digest differs, and ask again with the new facts shown.
  6. Show the reviewer the facts in the read set. An approval of a tool name is not an approval of an amount.
  7. Keep the interrupt node to the pause and pure computation. Put the re-read in its own node after the pause.
  8. Make the restore spec the input of the code that restores. A spec that only a startup check reads is a comment.
  9. Walk the compiled graph at startup, from every pause, and fail if any effect is reachable without passing the reconcile step. Then test the approval resume itself: fail if an observation was not re-read or a conclusion survived.
  10. Bound the void loop. After a second void on one ticket, hand it to a person.
  11. Close the last window at the system of record with a conditional write, or name the provider invariant you rely on. Idempotency keys only see your own repeats.
  12. Emit a signal when an approval is voided on evidence, with the field that moved, next to Part 6's decision records. A rising void rate means other writers are working the same tickets.

Harness Engineering book, Chapter 12: State, Checkpoints, Resume

This is Part 12 of a series that follows Harness Engineering for Production AI Systems one chapter at a time. Chapter 12, "State, Checkpoints, Resume," goes further into the store itself than this part does. It measures what four combinations of checkpointer and durability setting record for the same three nested counter increments, and only one of them gets it right. It also covers the effect log written before the call and the window it leaves, sizing and capping the thread store, durability per run mode, and how Temporal's programming model compares with a few hundred lines of harness code. Where the chapter refunds a whole invoice, this part works on Part 10's line-level signature, and it applies the chapter's restore rules at the approval resume, which the chapter's graph does not. Code for the chapter is in the ch12-state-checkpoints-resume tag of the companion repository.

References


Agentic AI

Follow for more technical deep dives on AI/ML systems, production engineering, and building real-world applications:


Books by Ranjan Kumar

Federated AI Agents cover

Federated AI Agents

Right-Sized AI cover

Right-Sized AI

Harness Engineering for Production AI Systems cover

Harness Engineering

The 7 GenAI Architectures cover

The 7 GenAI Architectures

Building Real-World Agentic AI Systems with LangGraph cover

Building Real-World Agentic AI Systems

The ChatML Handbook cover

The ChatML Handbook

The Chat Templates Handbook cover

The Chat Templates Handbook

Comments