The Oracle Problem: Why Most AI Agent Loops Fail

A team gave a coding agent one job: make the failing test suite pass. The loop was simple. Run the agent, run pytest, repeat until pytest exits 0. On the third iteration the suite went green and the loop reported success.

In review, someone noticed how. The agent had made the suite pass by commenting out the three failing assertions. The stopping condition was "tests pass." The agent satisfied it without doing the work.

The team reached for the fix almost everyone reaches for first. The model is not smart enough, so swap it for a bigger one. They did. The bigger model found a cleaner way to comment out the assertions. Same result. The model's willingness to game a weak check is real and documented, which is exactly why you cannot rely on the model not to. The leverage is in the loop: a stopping condition the agent cannot satisfy without doing the task closes the opening, whatever the model's disposition.

The thesis: agent reliability is decided in the loop, not the model

Here is the claim this article owns. On an agentic system, the reliability you want is decided in the loop, not in the model or the prompt. The loop is the code around the model: how many times it runs, what state it carries between runs, how it checks whether it is done, and what it does when a step fails. Teams keep tuning the model layer - a better prompt, a bigger model - when the defect sits in the loop layer. That reflex is the mindset shift many teams have not made: the system is the loop around the model, not the model alone. A smarter model does not fix a broken stopping condition. It satisfies it faster.

There is a sharper version of this claim, and it is the one the article owns. Every agent framework gives you a way to stop the loop. None gives you a way to know the loop is done. As of mid-2026, LangGraph caps a graph at 25 steps and the OpenAI Agents SDK caps a run at 10 turns. Those are termination, not completion. The check that decides whether the work is actually finished - the loop's oracle - is the one part no framework ships. You build it, every time, and it is the part that decides whether the agent is reliable.

One naming note before we go further. "Loop engineering" already has an owner. Cobus Greyling gave it a name in mid-2026 for the outer orchestration layer: scheduling, git worktrees, sub-agents, and the shift he summarises as "stop prompting, design the loop." That layer is real and it matters. This article is about the layer underneath it - the control loop inside a single agent turn, where the observe-reason-act cycle actually runs. That inner loop is where most "agent failures" are decided, and it is the part teams skip.

Why the agent loop dominates: the math is against you

The reason the loop dominates is arithmetic. An agent finishes a task as a sequence of steps, and each step can fail. If each step succeeds with probability p, a run of n steps succeeds end to end with probability p raised to the power n.

The numbers are worse than intuition suggests. A step reliability of 95%, which is high for a tool-using model, gives about 60% success over 10 steps and about 36% over 20. At 85% per step, 10 steps land near 20%. This is reliability in series, the same law that forces a manufacturing line with many parts to hold each part far above the target yield.

Read that the right way. A better prompt or model raises p, and because of the exponent that helps a lot: 0.98 per step gives about 82% over 10 steps, well above the 60% you get at 0.95. So raising p is worth doing. It is also not enough. Prompting moves the per-step number but leaves the exponential structure intact. The loop is where you change the structure: run fewer steps so n is smaller, add a verifier that catches a bad step before it compounds, add a retry that recovers instead of propagating the error. Those break the chain of independent successes that p raised to the power n assumes. Anthropic states the base case plainly in its guidance: an agent is "just" a model using tools "based on environmental feedback in a loop." Everything that makes that loop reliable is engineering you add around the model.

The cost side tells the same story. A loop with no ceiling and no progress check does not just fail. It fails expensively. It calls tools and burns tokens until something outside the loop stops it.

The oracle problem: why agent loops inherit software testing's hardest bug

Software testing has a name for the hard part, and it is not running the test. It is knowing whether the test passed. That is the oracle problem, studied for decades. A test oracle is the mechanism that decides pass or fail, and building a reliable one is the bottleneck that limits how far test automation can go. Barr and colleagues survey the problem in depth: the oracle, not the test runner, is where the difficulty lives.

Agent loops inherited this problem whole. A framework gives you the loop and its termination. It will stop your agent after a set number of steps. It does not give you the oracle. Nothing in LangGraph or the OpenAI Agents SDK knows whether your agent actually finished the task. They know only that it ran out of turns. The completion check is yours to write, every time.

This is the claim to hold onto. Termination is solved and shipped in every framework. Completion is not. A framework can ship the hook for a completion check - a conditional edge, an output-type guardrail - but not the criterion, because only you know what "done" means for your task. The socket is generic; the oracle is yours. And because a gameable or absent oracle lets every other guardrail pass a wrong result as a right one, the oracle is the part that decides whether the loop is reliable at all.

The wrong way: run until "done" and hope

Here is the default agent loop, written the way most first versions are written.

# The naive loop: run until "done", hope for the best.# model(), run_tool() are provided by the agent framework.def run_agent(task: str) -> str:    history = [{"role": "user", "content": task}]    while True:                                   # no ceiling        response = model(history)                 # one model call        history.append(response)        for call in response.tool_calls:            result = run_tool(call)               # side effects, no checks            history.append(result)        if "DONE" in response.text:               # gameable stop condition            return response.text

Four defects, all in the loop, none in the model:

• No ceiling. while True has no maximum step count and no budget. A stuck agent runs until you kill it. Hand-roll the loop like this and you do not even get the framework's ceiling for free.
• A gameable stop condition. The loop ends when the model writes "DONE." The model decides when it is finished, and it can write "DONE" whether or not the task is complete.
• No progress check. The loop cannot tell the difference between an agent making progress and an agent calling the same failing tool ten times.
• No error recovery. A tool result goes straight back into the history with no handling. A tool that errors just feeds the error back and invites the same call again.

Swapping model for a stronger model changes none of these. The failure survives the upgrade because the failure is not in the model.

The right way: make the missing decisions explicit

An engineered loop makes those decisions explicit. It bounds the work, carries state forward, checks progress separately from completion, verifies completion against an oracle the agent cannot fake, and recovers from a failed step.

import hashlibfrom dataclasses import dataclass, field# Framework primitives: model(), run_tool(), observe(), ToolError.# You write these - they are task-specific, and no framework ships them:#   task_is_complete() is the oracle; summarize() and escalate() are your#   completion-summary and handoff policy.@dataclassclass LoopState:    history: list = field(default_factory=list)    steps: int = 0    tokens: int = 0    recent_actions: list = field(default_factory=list)  # for no-progress detectiondef action_signature(call) -> str:    # Same tool + same args = same action. Repeats signal a stuck loop.    raw = f"{call.name}:{sorted(call.arguments.items())}"    return hashlib.sha256(raw.encode()).hexdigest()def run_agent(task: str, state: LoopState,              max_steps: int = 15, token_budget: int = 200_000) -> str:    state.history.append({"role": "user", "content": task})    while state.steps < max_steps and state.tokens < token_budget:   # ceiling        state.steps += 1        response = model(state.history)        state.tokens += response.usage.total_tokens   # model tokens only        state.history.append(response)        for call in response.tool_calls:            sig = action_signature(call)            if state.recent_actions[-8:].count(sig) >= 2:            # no-progress                state.history.append(observe(                    "Repeated action, no new result. Try a different approach."))                continue            state.recent_actions.append(sig)            try:                result = run_tool(call)                              # error recovery            except ToolError as e:                result = observe(f"Tool failed: {e}. Recover or pick another tool.")            state.history.append(result)        # In production, persist `state` to a store here and rehydrate it next        # turn - that is the fix for state amnesia (failure mode 3).        if task_is_complete(task, state):        # the oracle - YOU write this            return summarize(state)    return escalate(state)   # ceiling hit: hand off, do not claim success

The stop condition is the important change. task_is_complete does not ask the model whether it is done. It runs an external check the agent does not control: the real test suite in a clean checkout, a schema validator, an exit code from a process the agent did not start. That check is the loop's oracle. It is a gate on completion, the same shape as a gated execution check on an agent's actions. If the oracle is gameable, the loop is broken no matter how good the rest is.

Note that the two exits are not the same. Reaching completion returns a summary. Reaching the ceiling calls escalate, which hands the task to a human or a fallback and reports that it did not finish. A loop that hits its limit and reports success is lying to you.

Here is the same loop as a picture. It has two nested loops: the outer path is the iteration ceiling (the while), and the inner path runs the tool calls in one model response (the for). The oracle is checked once per iteration, after the inner loop. Every gate on it is a decision the naive loop skipped.

flowchart TD
    S([Task in]) --> CEIL{Ceiling left?<br/>steps and budget}
    CEIL -- no --> ESC([Escalate<br/>hand off, report not done])
    CEIL -- yes --> RSN[Reason: model call<br/>read carried state]
    RSN --> MORE{More tool calls<br/>in this response?}
    MORE -- yes --> PROG{Same action<br/>repeated?}
    PROG -- yes --> NUDGE[Nudge: force a<br/>different approach]
    NUDGE --> MORE
    PROG -- no --> ACT[Act: run the tool]
    ACT --> ERR{Tool failed?}
    ERR -- yes --> REC[Recover: feed<br/>the error back]
    ERR -- no --> SAVE[Record result]
    REC --> MORE
    SAVE --> MORE
    MORE -- no --> ORACLE{External oracle:<br/>task complete?}
    ORACLE -- yes --> DONE([Return verified result])
    ORACLE -- no --> CEIL

    classDef start fill:#FFD93D,color:#2C2C2A,stroke:#D4B02A;
    classDef step fill:#4A90E2,color:#FFFFFF,stroke:#3A7BC8;
    classDef gate fill:#7B68EE,color:#FFFFFF,stroke:#6858DE;
    classDef good fill:#6BCF7F,color:#2C2C2A,stroke:#4CAF64;
    classDef bad fill:#E74C3C,color:#FFFFFF,stroke:#B03A2E;

    class S start;
    class RSN,NUDGE,ACT,REC,SAVE step;
    class CEIL,MORE,PROG,ERR,ORACLE gate;
    class DONE good;
    class ESC bad;

The diagnostic: loop-layer versus model-layer debugging

The split between the loop layer and the model layer is an idea the field started naming in 2026. This article sharpens it into a debugging rule you run before every model swap.

When an agent misbehaves, the first question is not "which model." It is "which layer."

A model-layer defect is a genuine reasoning failure. The model cannot do the task even with a perfect loop, carried state, and a fair oracle. A loop-layer defect is everything else. The model could do the task, but the loop let it stop early, lose state, repeat itself, or run forever.

Loop-layer defects are the more common class in my experience, and the more overlooked. They get misdiagnosed because the symptom always shows up at the model's output. The agent produced a wrong result, so the model looks guilty. But the wrong result came from a loop that fed the model stale state, or accepted a fake completion, or never let it recover from a failed step. Telling the two apart in production takes observability that traces which layer failed, not just the model's final output.

The test is cheap. Before you touch the model, ask one question: if I fixed the loop - a real oracle, carried state, progress detection, a sane ceiling, error recovery - would this failure survive? If the answer is no, it is a loop-layer defect. A bigger model will not fix it. It will reach the same wrong result more fluently.

The five loop-layer failure modes

The engineered loop above has work to do beyond calling the model: stop at the right time, carry state, detect progress, and recover from failure. Stopping is two jobs, not one - terminate on a ceiling and complete on an oracle - which is why these concerns produce five failure modes. Each mode is one of them left undone. Here they are as one reference, followed by the detail.

1. The gameable oracle

A gameable oracle is a stopping condition the agent can satisfy without doing the work. The opening story is the pure case: "tests pass" is gameable, because the agent can edit the tests. This is the loop-layer face of a well-studied problem - specification gaming, also called reward hacking, where an agent optimises the measurable proxy instead of the goal it stands for. It is documented for frontier models: they will delete or weaken tests to make a suite go green, a result I cover in detail in the Ralph Loop and /goal.

The fix is an oracle the agent does not control and cannot edit. Run the tests in a clean checkout the agent cannot touch. Validate the output against a schema the agent did not write. Check an exit code from a process the agent did not start. The harder the oracle is to fake, the more the loop's "done" means what you think it means.

Sometimes a perfect oracle is impossible, because the task has no single correct output to check against. Testing hit this wall long ago and answers it with partial oracles: checks weaker than a golden answer but still hard to fake. Two kinds transfer well. An invariant checks a property any correct output must hold: a summary must be shorter than its input and must not name an entity that was absent from the source. A metamorphic relation checks that a transformed input changes the output in a required way: a refactor must leave the test suite's behaviour unchanged, and the same question asked two ways must return the same answer. Both are harder to game than a self-reported "done," because the agent cannot satisfy a real property by editing the check. It would have to make the property actually hold.

2. The progress-termination gap

A loop usually knows how to terminate. It rarely knows whether it is progressing. Those are different questions, and most loops answer only the first.

The gap shows up at the framework defaults. As of mid-2026, LangGraph stops a graph after 25 super-steps by default and raises GraphRecursionError; the OpenAI Agents SDK stops after 10 turns by default and raises MaxTurnsExceeded. Those ceilings are termination, not progress. An agent can burn all 25 steps repeating one failing action, and the only signal you get is the error at the wall. Progress detection closes the gap. Hash the action and its arguments, and if the same action repeats with no new result, break the loop or force a different approach. That is the recent_actions check in the engineered loop above.

3. State amnesia across iterations

State amnesia is different from context rot. Context rot is too much in the window at once; the model loses the signal in the noise. State amnesia is the opposite failure: the loop never carried the state forward at all, so each iteration starts blind to what the last one learned.

The fix is externalised state. Write what matters to a file or a store between iterations, and read it back at the start of the next one. Anthropic's guidance for long-running agents uses exactly this pattern - a progress file plus git history that survives a context-window reset. I treat this as its own harness dimension in state management for agentic systems, because carrying state across steps is a distinct problem from managing what sits in the context window on a single step.

4. No ceiling

A loop with no ceiling does not just fail. It fails expensively. Every loop needs two ceilings: a step count and a budget in tokens or spend. The engineered loop above checks both in its while condition.

These ceilings are blunt. They do not know whether the work was useful, and a legitimate long task can hit them. That is fine. They are the backstop, not the strategy. Their job is to bound the worst case, not to decide when the agent is done - that is the oracle's job.

5. No error recovery

The naive loop appends a tool result to the history whether the tool succeeded or failed. A tool that errors just feeds the error back, and the agent, seeing the same state, tends to make the same call again.

The fix is to treat a tool failure as an observation the model must respond to, not a silent append. Catch it, describe what failed, and let the next step choose to recover or pick a different tool. This is where retry, fallback, and circuit-breaking belong - inside the loop, as first-class steps, a topic I cover in retry, fallback, and circuit-breaking for LLM resilience. A verifier or critic gate lives here too: a cheap second check that catches a bad step before it compounds, in the spirit of the Reflexion self-reflection loop, and close to a validation layer that repairs bad output.

Control theory gives you the vocabulary, not the guarantees

There is an older field that spent decades on loops that must stay stable and reach a goal: control engineering. The mapping to agent loops is real, and it is useful as vocabulary.

A single-shot prompt is open-loop control: one action, no feedback. ReAct - the reason-and-act pattern from Yao and colleagues - and Reflexion, which adds a self-reflection step, are closed-loop control: the environment's response, a tool result or an error, feeds back and changes the next action. The borrowed words earn their place. Feedback, convergence, oscillation, and termination all name real behaviours of an agent loop, and having the names makes the failures easier to see. The progress-termination gap is just oscillation the loop cannot detect.

One honest limit. Control theory also has formal results: provable stability, guaranteed convergence. Those do not transfer to a language-model loop. An agent loop is not provably convergent, and treating it as if it were is a mistake. Use control theory for its vocabulary and its discipline, not for guarantees it cannot give you here.

Loop engineering vs. harness engineering: where the inner control loop fits

Greyling's loop engineering is the outer loop: the system that discovers work and hands tasks to agents, with scheduling, worktrees, and sub-agents. That layer is worth building. But it assumes the inner loop works. If the control loop inside a single agent turn has a gameable oracle and no progress detection, a better outer loop just runs a broken inner loop more often, on a schedule.

That is why I treat the inner loop as part of the harness - the control-flow slice of it. It is the same argument I make across the Harness Engineering series and apply to real code in Claude Code on brownfield codebases: the intelligence is in the model, but the reliability is in the layer you build around it. The loop is the most control-flow-heavy part of that layer.

The loop engineering checklist

Before you ship an agent loop, or when one misbehaves, run these questions in order:

1. Oracle. Is the agent's stopping condition an external check it cannot edit? If the agent can make "done" true without doing the work, nothing else matters.
2. Progress. Does the loop detect no-progress separately from termination? A step ceiling is not progress detection.
3. State. Is state carried across iterations in something outside the context window, so the next step is not blind?
4. Ceilings. Does the loop have both a step ceiling and a budget ceiling as a backstop?
5. Recovery. Does a tool failure become an observation the model must handle, rather than a silent append that invites the same call?
6. Honest exit. On hitting the ceiling, does the loop escalate and report that it did not finish, rather than claim success?
7. Diagnosis. When it fails, do you debug the loop layer before the model layer?

If you can add only one thing, add the external oracle. A completion check the agent cannot game is the single highest-leverage part of the loop. Everything else bounds the damage; the oracle decides whether "done" is true. This is the oracle problem, and no framework will solve it for you - it is the part of the loop you own.

The model you are running is probably not your bottleneck. The next time an agent fails, resist the swap to a bigger model until you have asked whether the loop let it fail. Most of the time it did. The intelligence is in the model. The reliability is in the loop. Engineer the loop.

References

• Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2022). ReAct: Synergizing Reasoning and Acting in Language Models. arXiv:2210.03629. https://arxiv.org/abs/2210.03629
• Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., & Yao, S. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. NeurIPS 2023. arXiv:2303.11366. https://arxiv.org/abs/2303.11366
• Barr, E. T., Harman, M., McMinn, P., Shahbaz, M., & Yoo, S. (2015). The Oracle Problem in Software Testing: A Survey. IEEE Transactions on Software Engineering, 41(5), 507-525. https://ieeexplore.ieee.org/document/6963470
• Anthropic. (2024). Building Effective Agents. https://www.anthropic.com/research/building-effective-agents
• Anthropic. (2025). Effective Harnesses for Long-Running Agents. https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents
• Greyling, C. (2026, June 9). Loop Engineering. Substack. https://cobusgreyling.substack.com/p/loop-engineering (repository: https://github.com/cobusgreyling/loop-engineering)
• LangChain. (2026). GraphRecursionError - recursion limit (default 25). LangGraph Documentation. https://docs.langchain.com/oss/python/langgraph/errors/GRAPH_RECURSION_LIMIT
• OpenAI. (2026). Agents SDK - Running agents (max_turns, default 10). https://openai.github.io/openai-agents-python/running_agents/
• Huntley, G. (2025). Inventing the Ralph Wiggum loop. Dev Interrupted interview. https://devinterrupted.substack.com/p/inventing-the-ralph-wiggum-loop-creator

Related Articles

Agentic AI

• The Ralph Loop and /goal: What Claude Code Actually Automated
• Agentic AI in the Data Center Boom: A Unified Map of Where Agents Actually Run the Building
• Claude Code Guide: Build Agentic Workflows with Commands, MCP, and Subagents
• Claude Code on Brownfield Codebases: The Harness Is the Product