At 02:40 the billing agent's primary model started returning 529, overloaded, and kept doing it for twenty-six minutes. The gateway did what it was configured to do. It retried twice, switched to LLM fallback models from two other vendors, and returned a result for every request. In that window the queue-drain run worked through 212 tickets. Zero failed requests showed on the dashboard, and nobody was paged.
In the morning three things did not add up. Fourteen refunds had gone out during the window, issued by models nobody had ever watched issue a refund. Throughout, the run held the authority to move money that the team had granted after months on the primary. A request limit of 20 was meant to bound each run, and it never fired: the highest count it showed on any ticket was 12. Yet the provider logs showed tickets that had sent 28 requests.
And a customer on account acct_4417, owed 120.00 for a duplicated SSO charge on invoice inv_9107, had received this reply: "I'm not able to process that change right now, but any eligible adjustment will appear on your next invoice." It passed the schema check. It is a refusal written as an answer, and the harness had no way to read it as one, because the primary refuses in a different shape. (I constructed the night from the billing support agent this series is built around. Every gateway mechanic in it is documented behaviour, cited below.)
Nothing failed, so nothing alerted. The fourteen refunds were for the right amounts, because the refund checks read the ledger rather than the model. Nothing like that stood behind the replies. The run also held the authority to send replies to customers without review, granted on the same months of watching the primary, and the reply to acct_4417 went out under it. Every other control had been sized on the primary too: the run limit, the repair ladder behind the schema check, and the refusal handling. None of them had been measured on the models that were running.
My claim in this part is that falling back is a change of authority, and the harness should treat it as one. A fallback model is what the book calls the Second Plant: a different system, whose behaviour you have not characterised, so every bound you set from measurements of the primary is unmeasured on it. The authority the run holds is one of those bounds, and the one with the largest consequences. So a fallback is admitted by declaration when the breaker opens, before the run continues on it. Its window must clear the run's floor. Its declared band, the widest authority it may hold, must cover the run's. (Bands come from the Operational Authority Gradient, which the book adopts.) And a dated measurement of it must exist, including how it refuses, against which its own repair ladder and what is left of the run budget are checked. Otherwise it is refused, and the ticket waits or restarts at a narrower band.
Admission also needs a budget that is counted where the requests are sent. A fallback usually lives in a router or gateway that retries and falls back per request, and that is where the book's Budget Amnesia lives: a budget held by an object whose lifetime is shorter than the scope it is supposed to bound. Every check passes, and none of them bounds the run's total. A neighbouring gap already has a vendor fix: Pydantic AI Harness counts the fallback responses it rejected, which the core run counter omits. What is new here is the admission check. The fallback's repair ladder is checked against its own dated pass rate, and its worst case against what is left of the run budget.
Someone made the first half of that argument before me, and I should say who. Tian Pan wrote in May that "falling back is not the same as degrading gracefully", and that "A configured fallback proves that your router works. It proves nothing about whether your application works on the fallback's output" (Pan, 2026a). What this part adds is the admission check that acts on it, built around a question I did not find any gateway, SDK or agent framework asking: may this model hold the authority this run already has?
It also reverses my own April post on this layer, which said "A fallback that serves degraded-but-functional responses is better than rejecting requests" (retry, fallback and circuit breaking). For a model that has been measured, that is often still true. For one that has not, "degraded" describes the uptime and says nothing about the answers. My June post already argued that failures need classifying before anything retries, that a fallback needs a wall-clock deadline rather than an attempt count, and that the SDK's own retries should be turned off when you own the policy (fault isolation and circuit breaking). This part does not repeat any of that.
Why an LLM fallback model is not a degraded mode
Part 2 borrowed a word from control engineering. The plant is the thing being controlled, and the book's Plant Model is a dated measurement of how your model behaves on your own inputs, which expires when the model id, the serving platform or stack behind it, its sampling settings or your prompt template changes. Part 2 also noted in passing that a separate screening model is "a second plant" with its own entry. A fallback is the same case at the worst time: during an incident, the harness keeps its controls and swaps the thing they were designed around.
Not every control cares. The useful move is to split the harness into the controls whose correctness does not depend on which model is running, and the ones whose settings were chosen from a measurement of the primary.
grid-rows: 1
grid-gap: 30
hold: "Holds on any plant" {
label.near: top-center
style: {fill: "#EAF7EE"; stroke: "#3F9E52"; font-color: "#2C2C2A"}
grid-columns: 1
grid-gap: 14
a: "refund comparator reads the ledger\n(Part 1)" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
b: "duplicate cap reads the charge rows\n(Part 2)" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
c: "floor check runs per call,\nagainst that call's model (Part 5)" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
}
sized: "Sized on the primary" {
label.near: top-center
style: {fill: "#FDECEA"; stroke: "#C0392B"; font-color: "#2C2C2A"}
grid-columns: 1
grid-gap: 14
d: "re-asks on the Repair Ladder:\nset from first-attempt pass rate (Part 7)" {style: {fill: "#E74C3C"; stroke: "#C0392B"; font-color: "#FFFFFF"}}
e: "run budget: set from calls per ticket\non the primary" {style: {fill: "#E74C3C"; stroke: "#C0392B"; font-color: "#FFFFFF"}}
f: "refusal handling: built for the\nprimary's explicit refusals" {style: {fill: "#E74C3C"; stroke: "#C0392B"; font-color: "#FFFFFF"}}
g: "authority band the run holds:\ngranted on the primary's record" {style: {fill: "#E74C3C"; stroke: "#C0392B"; font-color: "#FFFFFF"}}
}
The left column is what Part 2 meant by keeping adversarial decisions in code that reads a system of record. Part 1's refund comparator and the duplicate cap read the ledger. They give the same verdict whichever model proposed the refund, so the 120.00 customer could not have been refunded 960.00 that night. Part 5's floor check also holds, because it reads the profile of the model each call actually uses. On a fallback with a smaller window it refuses at assembly, which Part 5 showed on an 8,192-token deployment.
The right column is where the Second Plant does its damage. None of those numbers is wrong for the primary; each one came from watching it. Part 7's ladder of 2 re-asks is ample because the primary's first attempt passes the shape check almost every time. A ticket on the primary takes 4 to 6 calls, so the run budget is 20. Refusal handling looks for an explicit refusal, because that is how the primary refuses. And the run holds the authority to issue refunds because the team watched the primary handle refunds for months. A fallback inherits all four settings and none of the evidence behind them. The differences are measured, not hypothetical. OR-Bench found over-refusal rates varying widely across 32 models from 8 families (Cui et al., 2025), and JSONSchemaBench found structured-output engines, two vendors' APIs among them, differing twofold in the real-world schemas they support (Geng et al., 2025). A cross-vendor fallback changes that engine too.
Same weights on the same platform are a different case. A failover to the same model id in another region or replica of the same pinned deployment is the same plant on another machine. It still needs the floor check, because deployments can cap context differently, but it does not need a new measurement. The same id on another platform does: Part 2's Plant Model expires when the serving platform or endpoint changes, and a model group in a gateway is an alias that can hold different ids altogether. If your measurements show regions differing, record the region as the platform and the exemption narrows with it. One caveat from LiteLLM's own docs: a fallback "to a deployment on another provider, or on another API key, cannot decrypt the encrypted reasoning the failed deployment produced on earlier turns", so the target "reasons fresh on that turn" (LiteLLM, reliability). Same weights, less state. A fallback chain also shares more fate than it looks, which the deterministic floor post argued for chains in general.
What LiteLLM, OpenRouter, Portkey and Anthropic check before they fall back
Gateways are better than the April post assumed, so it is worth being exact about how. I read the current docs for four of them, for the fallback built into LangChain's runnables, and for the one in the Claude Agent SDK.
| Fallback | Triggers on | Checks about the fallback before switching | Tells you it switched |
|---|---|---|---|
| LiteLLM Router | after num_retries; separate lists for context-window and content-policy errors | with enable_pre_call_checks, drops deployments whose window is too small for these messages | attempted_fallbacks in spend logs |
OpenRouter models | "any error", including "Moderation flags" and "Context length validation errors" | none documented | the response's model field |
| Portkey | "any non-2xx status code" by default | none; the docs say "Ensure fallback LLMs are compatible with your use case" | gateway logs |
LangChain with_fallbacks | exceptions_to_handle, default (Exception,) | none | only if you trace it |
Claude Agent SDK fallback_model | the primary "is overloaded or unavailable"; the primary is retried at the start of each user turn | none documented; tools and permissions are set on the session, not per model | the model field on each assistant message |
| Anthropic server-side fallback (beta) | a refusal, inside one API call | by default the model Anthropic recommends for the refusal category; or up to three of yours, each one of the requested model's published allowed_fallback_models | a fallback_message entry in usage.iterations |
Sources: LiteLLM routing, OpenRouter model fallbacks, Portkey fallbacks, LangChain with_fallbacks, Claude Agent SDK, configuration, Anthropic, refusals and fallback.
LiteLLM and Anthropic deserve credit. LiteLLM's pre-call check is a real admission check, and the docs are clear that without it "requests are sent to the provider regardless of input token count". But it asks whether this prompt fits. Your floor asks whether the spans this run cannot proceed without will fit, and only your harness knows those. Anthropic's allow-list is a declared set of admissible fallbacks, the provider's answer to "may this model serve this refused request". Neither asks whether the fallback may hold your run's authority, and I did not find a gateway, SDK or agent framework that does. The Agent SDK row is the plainest case. Its docs describe the session switching to the backup and document no check on it, and because tools and permissions are configured on the session rather than per model, the backup inherits them. That question does not belong in a gateway's defaults. A gateway knows the run's authority only if you send it, and the check also needs the run's floor and the fallback's measurements, which live in your harness.
LiteLLM, OpenRouter and Anthropic can also fall back on refusals or moderation flags. That sends a request one model declined to a model more likely to comply. For a billing agent that is an authority decision, and the harness should make it on purpose.
Budget Amnesia: why the run limit never saw the retries
The book defines Budget Amnesia as a budget held by an object whose lifetime is shorter than the scope it is supposed to bound. Held means owned: the object that constructs the count. Each object checks its own limit, never comes close, and is thrown away. None of them bounds the run, which is the scope the limit was meant for.
An agent run has at least four lifetimes nested inside each other, and each common limit lives at one of them.
direction: down
run: "run: one ticket, minutes" {
label.near: top-center
style: {fill: "#E8F1FB"; stroke: "#2C6FB0"; font-color: "#2C2C2A"}
grid-columns: 1
grid-gap: 18
turn: "turn: one model step" {
label.near: top-center
style: {fill: "#EEEAFB"; stroke: "#5A4BC4"; font-color: "#2C2C2A"}
grid-columns: 1
grid-gap: 18
call: "call: one request through the router" {
label.near: top-center
style: {fill: "#FFF6D6"; stroke: "#C9A227"; font-color: "#2C2C2A"}
grid-columns: 1
grid-gap: 14
att: "attempt: one HTTP request\nto one deployment" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
rlimit: "router: num_retries,\nthen fallbacks" {style: {fill: "#E74C3C"; stroke: "#C0392B"; font-color: "#FFFFFF"}}
}
ladder: "repair ladder: re-asks\nper validation (Part 7)" {style: {fill: "#E74C3C"; stroke: "#C0392B"; font-color: "#FFFFFF"}}
client: "client: request limit 20,\nbuilt again every turn" {style: {fill: "#E74C3C"; stroke: "#C0392B"; font-color: "#FFFFFF"}}
}
limit: "run: nothing here\noutlives a turn" {style: {fill: "#95A5A6"; stroke: "#6C7A7B"; font-color: "#2C2C2A"}}
}
Each red limit forgets when its container closes. When the request returns, the router's attempt counter is gone, and every validation restarts the ladder's count. The request limit of 20 is the one meant to bound the run, but it lives on a client that the turn builds and throws away, so it starts again from zero four times per ticket. Nothing in the run box outlives a turn.
Why per-call retry limits miss the run's total requests
Here is that arrangement with the provider scripted, so the run is reproducible. The primary is overloaded and refuses every request. The fallback always answers, and whether its JSON passes the shape check follows a fixed list.
from dataclasses import dataclass, fieldfrom datetime import dateclass Overloaded(Exception): """A 529: the deployment refused before doing any work."""@dataclassclass Provider: """Stands in for two real deployments during an outage. The primary is overloaded. The fallback answers every request, and whether its JSON passes the shape check follows a fixed script, so every run below is reproducible. """ fallback_parses: list[bool] sent: list[str] = field(default_factory=list) # every request, any model def call(self, model: str) -> bool: self.sent.append(model) if model == "primary": raise Overloaded(model) return self.fallback_parses.pop(0) # True: the output parsedEach model gets 1 + num_retries attempts before the router moves down the list, the way LiteLLM's Router does: "If a call fails after num_retries, LiteLLM falls back to another model group" (LiteLLM, reliability). On a failed shape check the run re-asks, as Part 7's Repair Ladder does for shape failures. The client carries the request limit of 20, and the turn builds a new client each time, which is what a graph node that constructs its own client does on every invocation. PydanticAI's own limit behaves this way when a graph node calls agent.run(..., usage_limits=UsageLimits(request_limit=20)) on each invocation: the limit is per run(), and the node starts a new one every turn.
class Router: """A gateway-style client: retries, then falls back, per request. It carries a request limit meant to bound the run. The count lives as long as this object does. """ def __init__(self, provider, models, num_retries, request_limit): self.provider = provider self.models = models self.num_retries = num_retries self.request_limit = request_limit self.sent = 0 def complete(self) -> bool: for model in self.models: for _ in range(1 + self.num_retries): if self.sent >= self.request_limit: raise RuntimeError(f"limit of {self.request_limit} reached") self.sent += 1 try: return self.provider.call(model) except Overloaded: continue raise RuntimeError("every model failed")def run_ticket(provider, turns, reasks): """One client per turn, the way a graph node often builds its own.""" peak = reasked = 0 for turn in range(1, turns + 1): router = Router(provider, ("primary", "fallback"), num_retries=2, request_limit=20) # Part 7's narrow rung is left out here; it adds one request per turn for tries in range(1 + reasks): # re-ask on a failed shape check if router.complete(): reasked += tries break else: return f"turn {turn} deferred after {1 + reasks} tries", peak, reasked peak = max(peak, router.sent) return "finished", peak, reaskedSCRIPT = [False, True, False, False, True, True, True] # fallback parsesprovider = Provider(fallback_parses=list(SCRIPT))outcome, peak, reasked = run_ticket(provider, turns=4, reasks=2)print("a request limit of 20 on a client built per turn")print(f" outcome: {outcome}")print(f" highest count any limit saw: {peak} of 20")print(f" requests sent: {len(provider.sent)} " f"({provider.sent.count('primary')} to the primary, " f"{provider.sent.count('fallback')} to the fallback)")print(f" re-asks after a failed shape check: {reasked}")a request limit of 20 on a client built per turn outcome: finished highest count any limit saw: 12 of 20 requests sent: 28 (21 to the primary, 7 to the fallback) re-asks after a failed shape check: 3Every limit passed: the router never went past its 3 attempts per model, the ladder never past its 2 re-asks, and no client got past 12 of its 20. Yet the run sent 28 requests, more than the limit was meant to allow for the whole ticket, and no check saw more than 12 of them. Each object was well inside its limit, because each one lived for a single turn.
A provider may not bill the 21 requests to the primary, because they were refused before any work. They still cost the run wall-clock time, and they still land on a provider that is shedding load. With a few hundred runs in flight they are the retry amplification the June post described. Most of the 28 is that amplification, and the breaker below removes it. The share that belongs to the Second Plant is smaller and does not go away: all 7 requests sent to the fallback were answered and billed, and 3 of them were re-asks. At the primary's first-attempt pass rate of 0.97, four turns would usually have needed none.
Where retries stack in layers they multiply, which the June post covers. LiteLLM pins the provider client to max_retries: 0 for a call through its router for that reason (LiteLLM routing), while the Anthropic SDK used directly retries twice by default (Anthropic, Python SDK). Your run loop and your repair ladder are layers no gateway can see.
Retry budgets are old, and the well-known ones solve a different problem. The Site Reliability Engineering book's chapter on overload describes "a per-client retry budget" (Beyer et al., 2016), where a request is retried only while retries are under 10% of the client's requests. Envoy's retry_budget caps concurrent retries at a share of active requests (Envoy), and gRPC pauses retries when its token count falls below half (gRPC retry guide). They protect a backend from a fleet's retries, which is the breaker's job here, and they live as long as the client process, not as long as one run. Tian Pan makes the agent version of the point: "enforce the budget inside the loop, not outside", because "Most retry middleware sits at the HTTP layer, well below where the agent logic lives" (Pan, 2026b). Budget Amnesia names the failure underneath both: the limit is held by an object that does not live as long as the thing it is meant to bound.
A run-owned budget that every attempt spends
The fix is ownership. The run constructs one budget when it starts and passes it down. Nothing constructed per call or per turn owns a limit that must be counted per run; it may hold a reference to the run's. Before every request it sends, retries and fallback attempts included, the router spends from the run's budget.
direction: down
run: "run builds RunBudget(20)\nonce, at start" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
ladder: "repair ladder\nre-ask on a shape failure" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
router: "router\nretry, then fall back" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
spend: "budget.spend(model, attempt)\nbefore every request" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
req: "request to a deployment" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
stop: "BudgetExhausted:\nthe run stops and escalates" {style: {fill: "#E74C3C"; stroke: "#C0392B"; font-color: "#FFFFFF"}}
run -> ladder: "1. same budget object"
ladder -> router: "2. passed down"
router -> spend: "3. each attempt"
spend -> req: "4. under the limit"
spend -> stop: "at the limit"
The arrow that matters is the third. spend refuses before a request once the count is at the limit; record is for what the provider reports after a response, which has already happened and must be counted even past the limit. A retry and a fallback attempt go through the same spend as a first attempt, so the run sees them whatever the router decides.
class BudgetExhausted(Exception): passclass NoModelAvailable(Exception): pass@dataclassclass RunBudget: """A bound owned by the run. Every request any component sends spends it.""" requests_allowed: int sent: list[tuple[str, int]] = field(default_factory=list) # (model, attempt) def __post_init__(self): if self.requests_allowed < 1: raise ValueError("a run budget allows at least one request") def spend(self, model: str, attempt: int) -> None: if len(self.sent) >= self.requests_allowed: raise BudgetExhausted( f"{len(self.sent)} requests sent, limit {self.requests_allowed}") self.sent.append((model, attempt)) # one record per request def record(self, model: str, attempt: int) -> None: """After a response: an invocation the provider reports. Never refuses.""" self.sent.append((model, attempt))class BudgetedRouter: def __init__(self, provider, models, num_retries, budget): self.provider = provider self.models = models self.num_retries = num_retries self.budget = budget # passed down, never built here def complete(self) -> bool: for model in self.models: for attempt in range(1, 2 + self.num_retries): self.budget.spend(model, attempt) # before every request try: return self.provider.call(model) except Overloaded: continue raise NoModelAvailable("every model failed")def run_ticket_budgeted(budget, provider, models, turns, reasks): for turn in range(1, turns + 1): router = BudgetedRouter(provider, models, num_retries=2, budget=budget) for _ in range(1 + reasks): try: if router.complete(): break except BudgetExhausted as exc: return f"turn {turn}: budget exhausted, {exc}" except NoModelAvailable: return f"turn {turn}: deferred, no model available" else: return f"turn {turn} deferred after {1 + reasks} tries" return "finished"for label, models in (("no breaker", ("primary", "fallback")), ("breaker open", ("fallback",))): provider = Provider(fallback_parses=list(SCRIPT)) budget = RunBudget(requests_allowed=20) # once, for the whole run print(f"run-owned budget, {label}") print(f" outcome: " f"{run_ticket_budgeted(budget, provider, models, turns=4, reasks=2)}") print(f" requests sent: {len(provider.sent)}, " f"budget saw: {len(budget.sent)}")run-owned budget, no breaker outcome: turn 3: budget exhausted, 20 requests sent, limit 20 requests sent: 20, budget saw: 20run-owned budget, breaker open outcome: finished requests sent: 7, budget saw: 7With no breaker, the same ticket now stops at turn 3 with the reason attached. The budget's count matches the provider's here by construction, because the router spends before each request rather than after each answer; the agent-framework section below lists the places where a real stack breaks that match. Each spend is also the natural place to emit one record into Part 6's Second Stream, with the model and the attempt number, so the morning-after question "how many requests did this run send, and to which model?" has an answer that does not depend on the provider's invoice.
Stopping is the correct outcome, and an uncomfortable one. A run that sent 20 requests to finish half a ticket is one the harness should stop. Every run stopping at turn 3 during an outage is the book's hour of backlog again, though, and the breaker in the second run is what prevents it.
The second run shows what a breaker buys. Once the breaker is open, the router skips the primary, and the whole ticket costs 7 requests. A breaker removes the amplification. All three re-asks stay, because they are the price of running on a model that fails the shape check more often. And notice what the second run did not do: nothing asked whether this ticket should be running on the fallback at all. That is the next section.
Agent frameworks already have run limits. Check what they count.
Run budgets already exist. PydanticAI's UsageLimits caps tokens, requests and cost "for a single run()" (Pydantic AI Harness, spend), the Claude Agent SDK has max_budget_usd (configuration), the OpenAI Agents SDK raises MaxTurnsExceeded past max_turns (running agents), and LiteLLM has per-session caps for agents. A run limit is only as good as what it counts, though. Each documented gap below is a place where a fallback's extra work goes uncounted, or where a counter restarts in the middle of what you think of as one run.
- Rejected fallback responses. The Pydantic AI Harness docs say a response its
FallbackModelrejected "was billed as well", and that theirSpent.requests"counts these attempts, where core'sRunUsage.requestscounts only the responses the agent acted on" (Pydantic AI Harness, spend). That is a second failure, separate from the lifetime one above: the counter lives long enough, but it counts responses, not requests. The vendor documents it and ships a fix beside it. - Attempts that never produced a response. The Harness count above is described in terms of responses that were billed and rejected. A 529 returns no response, and the docs say an attempt that "raised before the provider answered reported no usage and is not charged". In the demo above, 21 of the 28 requests were of that kind, invisible to a counter of charged attempts. For a cost budget that is right; for a budget meant to bound load on a failing provider, it is the gap.
- Two models in one call. Anthropic's server-side fallback "retries a refused request inside a single API call", and you detect it from "A
fallback_messageentry inusage.iterations" (Anthropic, refusals and fallback). A budget that spends once per HTTP request undercounts by one every time this fires. After the response arrives,recordonce more for eachfallback_messageentry inusage.iterations. - Counters that restart. In the Claude Agent SDK with streaming input, "the max-turns count starts over for each queued message", although the dollar budget accumulates (Claude Agent SDK, configuration). LiteLLM's agent session counters "expire after 1 hour by default", and the session id is the one the caller sends (LiteLLM, agent iteration budgets). A queue-drain run that waits on a breaker for an hour gets a fresh count.
- Budgets that do not apply to the fallback. LiteLLM skips budget checks entirely for models priced at zero, and suggests that "as a fallback that budget-exhausted keys can still access" (LiteLLM, budgets). A self-hosted fallback configured at zero cost is outside every key budget you set.
Keep a second limit, measured in time. A run that retries with backoff spends most of its life waiting, so it also needs the wall-clock deadline the June post argued for. In the book's resilience config that is 20 model calls and 90 seconds per run. And a budget checked before each request cannot stop the request already in flight. Pydantic's docs say the same of their own ceiling: "The request that crosses the line completes". Size the limit so that one more request is still acceptable.
The Second Plant: an admission check for LLM fallback models
The book's definition is short: a fallback model "is not a degraded version of your system. It is a different plant, with a behaviour distribution you have not characterised, and every bound designed against the primary is undesigned against it." So the fallback has to declare what it still satisfies, and the harness checks that declaration when the breaker opens, before the run continues on it.
Six questions make up the check, ordered so the cheapest refusals come first.
grid-rows: 6
grid-columns: 2
grid-gap: 22
q1: "1. Window below the run's floor?" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
a1: "refuse: Part 5's floor, asked early" {style: {fill: "#E74C3C"; stroke: "#C0392B"; font-color: "#FFFFFF"}}
q2: "2. Run's band wider than declared?" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
a2: "refuse: authority does not transfer" {style: {fill: "#E74C3C"; stroke: "#C0392B"; font-color: "#FFFFFF"}}
q3: "3. Same id and platform as the primary?" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
a3: "skip measurement; budget still checked" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
q4: "4. Plant Model entry missing or stale?" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
a4: "refuse: nothing measured" {style: {fill: "#E74C3C"; stroke: "#C0392B"; font-color: "#FFFFFF"}}
q5: "5. Hedged refusals above propose-only?" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
a5: "refuse: no person reads the reply" {style: {fill: "#E74C3C"; stroke: "#C0392B"; font-color: "#FFFFFF"}}
q6: "6. Ladder, measured band, budget?" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
a6: "admit with its own ladder, or refuse" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
q1 -> q2
q2 -> q3
q3 -> q4
q4 -> q5
q5 -> q6
Read down the left column. The band question comes second, before anything about measurement. A fallback declared for drafting is refused for a refund run however well it was measured, and that includes a same-weights deployment that someone declared read-only. Only the third row skips the measurement, and only because the weights and the platform are the same; its budget is still checked.
Plain integers stand in for two richer objects. floor_tokens and window_tokens replace Part 5's ContextBudget and ModelProfile. In a harness that has those, the first question is Part 5's floor check, called with the fallback's profile when the breaker opens rather than with the primary's at assembly. turns_left is the run's remaining turn cap, its maximum turns minus the turns taken, not a forecast; the worst case is a bound because the cap is one.
BANDS = ("read-only", "propose-only", "reversible-writes", "irreversible-writes")SHAPES = ("explicit", "hedged")MAX_REASKS = 5 # past this, re-asking is not a repair planNARROW = 1 # Part 7's narrow rung: one more request per turndef check_band(band: str) -> str: if band not in BANDS: # a typo fails at load, not mid-incident raise ValueError(f"unknown band {band!r}") return banddef wider(band: str, than: str) -> bool: return BANDS.index(band) > BANDS.index(than)@dataclass(frozen=True)class PlantEntry: """A dated measurement of one model, as served, on your tickets (Part 2).""" model_id: str serving: str # the platform or endpoint it was measured on measured_on: date valid_days: int template: str # the prompt template it was measured with measured_band: str # widest band its shadow runs covered shape_pass_rate: float # first attempt, on your schema refusal_shape: str # "explicit" or "hedged" def __post_init__(self): if self.valid_days < 0: raise ValueError(f"valid_days {self.valid_days} is negative") check_band(self.measured_band) if not 0.0 <= self.shape_pass_rate <= 1.0: raise ValueError(f"pass rate {self.shape_pass_rate} is not a rate") if self.refusal_shape not in SHAPES: raise ValueError(f"unknown refusal shape {self.refusal_shape!r}")@dataclass(frozen=True)class Fallback: to: str model_id: str serving: str window_tokens: int max_band: str # the widest band it may hold, as declared plant: PlantEntry | None reasks: int | None = None # None: inherit the ladder sized on the primary retries: int = 0 # router retries on this model, per request def __post_init__(self): check_band(self.max_band) if self.reasks is not None and not 0 <= self.reasks <= MAX_REASKS: raise ValueError(f"reasks {self.reasks} is outside 0..{MAX_REASKS}") if self.retries < 0: raise ValueError(f"retries {self.retries} is negative")@dataclass(frozen=True)class RunPolicy: primary_model_id: str primary_serving: str template: str reasks: int # repair ladder, sized on the primary max_unrepaired: float # share of turns allowed to exhaust the ladder def __post_init__(self): if not 0 <= self.reasks <= MAX_REASKS: raise ValueError(f"reasks {self.reasks} is outside 0..{MAX_REASKS}") if not 0.0 < self.max_unrepaired < 1.0: raise ValueError("max_unrepaired must be between 0 and 1")@dataclass(frozen=True)class RunState: """What the run is, at the moment the breaker opens.""" band: str floor_tokens: int turns_left: int budget: RunBudget def __post_init__(self): check_band(self.band) if self.turns_left < 0: raise ValueError(f"turns_left {self.turns_left} is negative")class FallbackRefused(Exception): pass@dataclass(frozen=True)class Admission: to: str reasks: int # the ladder the run uses on the fallback worst_case: int # requests still to send, every turn to the last rung note: strdef reasks_needed(pass_rate: float, max_unrepaired: float) -> int | None: """Fewest re-asks r with (1 - p) ** (1 + r) at or under the target. Treats attempts as independent. They are not: a model that failed a shape once is likelier to fail it again. So this is a floor on the re-asks needed, never a ceiling. None: not within MAX_REASKS. """ for r in range(MAX_REASKS + 1): if (1 - pass_rate) ** (1 + r) <= max_unrepaired: return r return Nonedef fits_budget(fb: Fallback, run: RunState, ladder: int) -> int: worst = run.turns_left * (1 + ladder + NARROW) * (1 + fb.retries) left = run.budget.requests_allowed - len(run.budget.sent) if worst > left: raise FallbackRefused( f"worst case {worst} requests, {left} left in the run budget") return worstdef admit(fb: Fallback, policy: RunPolicy, run: RunState, today: date) -> Admission: if fb.window_tokens < run.floor_tokens: raise FallbackRefused( f"window {fb.window_tokens} is below the floor of {run.floor_tokens}") if wider(run.band, fb.max_band): raise FallbackRefused(f"run holds {run.band}; {fb.to} allows {fb.max_band}") if (fb.model_id, fb.serving) == (policy.primary_model_id, policy.primary_serving): worst = fits_budget(fb, run, policy.reasks) return Admission(fb.to, policy.reasks, worst, f"same weights and platform, worst case {worst} requests") p = fb.plant if p is None or (p.model_id, p.serving) != (fb.model_id, fb.serving): raise FallbackRefused(f"no Plant Model entry for {fb.to} on {fb.serving}") if p.measured_on > today: raise FallbackRefused(f"entry for {fb.to} is dated {p.measured_on}") if (today - p.measured_on).days > p.valid_days: raise FallbackRefused( f"entry for {fb.to} expired ({p.measured_on} + {p.valid_days} days)") if p.template != policy.template: raise FallbackRefused( f"entry for {fb.to} is for {p.template}, not {policy.template}") if p.refusal_shape != "explicit" and wider(run.band, "propose-only"): raise FallbackRefused(f"{fb.to} hedges its refusals; propose-only at most") ladder = policy.reasks if fb.reasks is None else fb.reasks need = reasks_needed(p.shape_pass_rate, policy.max_unrepaired) if need is None or need > ladder: raise FallbackRefused( f"pass rate {p.shape_pass_rate:.2f} needs " f"{need if need is not None else f'over {MAX_REASKS}'} re-asks; " f"the ladder allows {ladder}") if wider(fb.max_band, p.measured_band): raise FallbackRefused(f"{fb.max_band} declared, {p.measured_band} measured") worst = fits_budget(fb, run, ladder) return Admission(fb.to, ladder, worst, f"ladder {ladder}, worst case {worst} requests")The four bands come from the Operational Authority Gradient, which the book adopts, and Part 14 of this series builds them. Here the only property used is their order: read-only, then propose-only, then reversible writes, then irreversible writes such as issue_refund. A band name and a refusal shape are checked when the declaration is built, so a typo stops the deploy instead of surfacing during the incident, and an unknown refusal shape fails closed. The worst case is counted in the run budget's own unit, requests sent: every remaining turn taken to the last rung of Part 7's ladder (the first attempt, the re-asks and the one narrowed request), times the router's attempts per request on the fallback. It is compared with what is left of the run's budget, not the whole of it, because the breaker opens in the middle of a run. Ignoring what the run already spent would repeat Budget Amnesia inside the check that is meant to prevent it. Same weights skip the measurement, never the budget.
A declared band is not evidence either, or the check would only move the unearned authority into a config file. So the entry records measured_band: the widest band its own evidence covers, meaning tickets the fallback ran in shadow at that band, with the comparators agreeing and its refusals read and classified. A declaration may narrow that and never widen it. What the band protects is the writes no comparator reads: the reply sent to a customer, the decision to refund at all, an account change. A refund's amount has a ledger behind it. The decision to send acct_4417 a polite non-answer had nothing.
Here is the refund ticket from the opening, asked fourteen ways while the breaker is open. Model A is the primary, served on platform a-api. B is a second vendor's mid-size model, measured six weeks ago on this prompt template. C is a small model, measured in October, which refuses by hedging.
POLICY = RunPolicy(primary_model_id="vendor-a/large-2026-09", primary_serving="a-api", template="billing-v14", reasks=2, max_unrepaired=0.01)TODAY = date(2026, 11, 14)A, B, C = "vendor-a/large-2026-09", "vendor-b/medium-2026-08", "vendor-c/small-2026-10"WRITES, DRAFT, READ = "irreversible-writes", "propose-only", "read-only"def entry(model, serving, measured_on, template="billing-v14", band=DRAFT, rate=0.70, shape="explicit"): return PlantEntry(model, serving, measured_on, 90, template, band, rate, shape)b_plant = entry(B, "b-api", date(2026, 10, 2))b_v13 = entry(B, "b-api", date(2026, 10, 2), template="billing-v13")b_july = entry(B, "b-api", date(2026, 7, 1))c_plant = entry(C, "c-api", date(2026, 10, 20), rate=0.95, shape="hedged")b_draft = Fallback("b", B, "b-api", 200_000, DRAFT, b_plant, reasks=3)def fresh(band, spent=0): budget = RunBudget(requests_allowed=20, sent=[("primary", 1)] * spent) return RunState(band, floor_tokens=12_000, turns_left=4, budget=budget)cases = [ ("self-hosted, 8,192 window", Fallback("local", "local/8b", "local", 8_192, WRITES, None), fresh(WRITES)), ("B, never measured", Fallback("b", B, "b-api", 200_000, WRITES, None), fresh(WRITES)), ("B, measured in July", Fallback("b", B, "b-api", 200_000, WRITES, b_july), fresh(WRITES)), ("B, measured on v13", Fallback("b", B, "b-api", 200_000, WRITES, b_v13), fresh(WRITES)), ("B, measured at 0.70", Fallback("b", B, "b-api", 200_000, WRITES, b_plant), fresh(WRITES)), ("C, measured, hedges", Fallback("c", C, "c-api", 128_000, WRITES, c_plant), fresh(WRITES)), ("B, ladder 3, writes", Fallback("b", B, "b-api", 200_000, WRITES, b_plant, reasks=3), fresh(WRITES)), ("B, propose-only, ladder 3", b_draft, fresh(WRITES)), ("same, invoice-copy ticket", b_draft, fresh(DRAFT)), ("same, 9 requests spent", b_draft, fresh(DRAFT, spent=9)), ("same, router retries 2", Fallback("b", B, "b-api", 200_000, DRAFT, b_plant, reasks=3, retries=2), fresh(DRAFT)), ("A, same deployment pool", Fallback("region-2", A, "a-api", 1_000_000, WRITES, None), fresh(WRITES)), ("A on another platform", Fallback("a-cloud2", A, "cloud-2", 1_000_000, WRITES, None), fresh(WRITES)), ("A, declared read-only", Fallback("r3", A, "a-api", 1_000_000, READ, None), fresh(WRITES)),]print("the breaker is open. May this run continue on the fallback?")print(f" A at 0.97 needs {reasks_needed(0.97, 0.01)} re-ask; the ladder allows 2")for label, fb, run in cases: try: verdict = f"admitted: {admit(fb, POLICY, run, TODAY).note}" except FallbackRefused as exc: verdict = f"REFUSED: {exc}" print(f" {label:<26} {verdict}")for bad in (lambda: Fallback("b", B, "b-api", 200_000, "propose_only", b_plant), lambda: entry(C, "c-api", TODAY, shape="Hedged")): try: bad() except ValueError as exc: print(f" config typo {exc}")the breaker is open. May this run continue on the fallback? A at 0.97 needs 1 re-ask; the ladder allows 2 self-hosted, 8,192 window REFUSED: window 8192 is below the floor of 12000 B, never measured REFUSED: no Plant Model entry for b on b-api B, measured in July REFUSED: entry for b expired (2026-07-01 + 90 days) B, measured on v13 REFUSED: entry for b is for billing-v13, not billing-v14 B, measured at 0.70 REFUSED: pass rate 0.70 needs 3 re-asks; the ladder allows 2 C, measured, hedges REFUSED: c hedges its refusals; propose-only at most B, ladder 3, writes REFUSED: irreversible-writes declared, propose-only measured B, propose-only, ladder 3 REFUSED: run holds irreversible-writes; b allows propose-only same, invoice-copy ticket admitted: ladder 3, worst case 20 requests same, 9 requests spent REFUSED: worst case 20 requests, 11 left in the run budget same, router retries 2 REFUSED: worst case 60 requests, 20 left in the run budget A, same deployment pool admitted: same weights and platform, worst case 16 requests A on another platform REFUSED: no Plant Model entry for a-cloud2 on cloud-2 A, declared read-only REFUSED: run holds irreversible-writes; r3 allows read-only config typo unknown band 'propose_only' config typo unknown refusal shape 'Hedged'Begin with the second line of output, the bound that was sized on the primary. At a first-attempt pass rate of 0.97, one re-ask brings the chance of a turn exhausting the ladder to 0.03 squared, under 0.1%, so a ladder of 2 leaves room. Put B on the same ladder and the number changes. At 0.70, two re-asks leave 0.3 cubed, 2.7% of turns unrepaired, against a target of 1%. Nothing about the ladder changed, and it is now too short. That is "undesigned against it" as a number.
B measured in July is refused on age, and B measured on the previous prompt template is refused by Part 2's expiry rule, because a measurement of one template says nothing reliable about another. Of Part 2's five expiry triggers, the check enforces three, the model id, the serving platform and the template, along with the entry's age. The other two, a change to sampling settings and a change to the serving stack behind an id, are left to whoever owns the entry, because nothing in a request reveals them. C is the reply from the opening. A model that refuses by hedging produces text that passes a schema check, and unless you have a refusal comparator measured on that model, the control left is a person reading the draft. That works at propose-only, where a person reads every draft before it is sent. It does not work for a run that sends replies and moves money.
Expect arguments about the band rows. Give B its own ladder of 3 and declare it for irreversible writes, and it is still refused, because its entry was measured at propose-only. Declared honestly for propose-only, it is refused for the refund ticket on the band itself: the run holds irreversible-writes. For a ticket that only asks for a copy of an invoice, the same fallback is admitted: four turns of five requests each is a worst case of 20, exactly the run budget. If the run has already spent 9 requests on the primary before the breaker opened, only 11 are left, and the same fallback is refused. Give the router 2 retries per request on B and the worst case becomes 60, also refused. A fallback that is itself failing should stop the run, not triple it. The last two rows are the primary's id on another platform, refused for want of an entry of its own, and the same-weights deployment declared read-only, refused on its band like any other.
This check has limits. First, refusal_shape is a label standing in for a measured rate: in a real entry it is the share of refusals that arrive hedged, with a threshold above which the fallback is held at propose-only. Second, reasks_needed treats attempts as independent, and they are not: a model that broke the shape once is likelier to break it again. So 3 re-asks is a floor on what B needs, and the real rate on your traffic is the next measurement to take. Third, the book's companion code at the chapter tag declares only the window and the band for a fallback. Here the Plant Model, refusal-shape, ladder and budget checks extend the chapter; none of them is code from the tag. Fourth, the April post's FallbackRouter is the naive version of all this: it catches any exception from the primary and calls the secondary, with no admission step at all.
When an LLM fallback is refused: queue-drain vs copilot
When the admission check refuses and the primary is down, no system is available to run the refund ticket. So it waits, which is the honest outcome. That will be unpopular, because teams configure a fallback precisely so that nothing waits.
What waiting costs depends on the mode the agent runs in.
grid-rows: 1
grid-gap: 30
drain: "Queue-drain: refused" {
label.near: top-center
style: {fill: "#E8F1FB"; stroke: "#2C6FB0"; font-color: "#2C2C2A"}
grid-columns: 1
grid-gap: 14
d1: "ticket stays in the queue,\nnot picked up" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
d2: "drains when the breaker closes" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
d3: "cost: latency only" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
}
copilot: "Copilot: refused for writes" {
label.near: top-center
style: {fill: "#FFF6D6"; stroke: "#C9A227"; font-color: "#2C2C2A"}
grid-columns: 1
grid-gap: 14
c1: "a person is already waiting" {style: {fill: "#FFD93D"; stroke: "#C9A227"; font-color: "#2C2C2A"}}
c2: "a new run at propose-only\nmay be admitted (B, ladder 3)" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
c3: "the person issues the refund" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
}
In queue-drain mode the refusal is cheap. A ticket that is never picked up stays a ticket. One that is picked up, runs out of options and escalates has become work for a person. That is exactly the book's opening incident: four minutes of someone else's outage became an hour of backlog, because every run started, failed and escalated. Stop picking up refund tickets while the breaker is open, and let invoice-copy tickets drain on the admitted fallback.
In copilot mode a person is already waiting, so waiting has a real cost. The admission check gives a narrower choice instead of none. A new run at propose-only can be admitted on B. It drafts, looks things up and proposes the refund, and the person issues it. The run that held irreversible-writes does not continue on the fallback with that authority. A narrower run that restarts mid-ticket must also read the ledger and Part 6's Second Stream for writes the first run already made, and propose only what is still owed, or it proposes a refund that was already issued. Whether to start the narrower run is a policy decision, and Part 14 owns the general version of it: when a run's authority is lowered, and who may restore it.
Someone will propose the shortcut during the incident: lower the floor, or widen B's declared band, for the next hour. It is one config change and everything works again. It also means running an unmeasured model at full authority during an incident, which is when a wrong refund is least likely to be noticed. If the refusal rate during outages is unacceptable, measure a fallback ahead of time. Assembled reports what that costs: "every prompt change requires evaluation against at least the first two providers in our fallback sequence". By their account that added 20 to 30 percent to prompt development time, in exchange for "99.97% effective uptime" across several provider outages (Wang, 2025). That is the Plant Model entry, paid for in advance, and a Second Plant you can trust costs less that way than finding out during the incident.
Server-side fallback and allow-lists are fallbacks too
Providers can now switch models for you. Anthropic's server-side fallback, in beta under the header server-side-fallback-2026-07-01, handles a refusal: "you receive a normal response, not an error, with stop_reason: "refusal"", and with fallback enabled the request is retried "inside a single API call" on another model, up to three of your own choosing, with "one response that names the model that answered" (Anthropic, refusals and fallback).
direction: down
harness: "harness: one request\nbudget.spend once" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
api: "" {
style: {fill: "#FFF6D6"; stroke: "#C9A227"; font-color: "#2C2C2A"}
first: "one API call, model A:\nrefusal (may be billed)" {style: {fill: "#E74C3C"; stroke: "#C0392B"; font-color: "#FFFFFF"}}
second: "fallback model: answer\n(billed)" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
first -> second: "inside the call"
}
check: "harness reads usage.iterations:\nrecord once more per fallback_message,\nrun admit() for the model named" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
harness -> api.first: "1. request"
api.second -> check: "2. one response"
For the harness, the budget has to spend on what comes back, not only on what was sent, because one request can carry two model invocations, and the docs say the triggering refusal is billed too in some cases. And every model on your fallback list is a declared fallback in the sense of the previous section. If it is allowed to answer a refund run, it needs a Plant Model entry and a band that covers that run. A refusal answered by a model more willing to comply is the case where the band check matters most.
This applies equally to content_policy_fallbacks in LiteLLM and to OpenRouter's default of falling back on moderation flags. A refused request is a decision by the primary, and passing it to a different model is a decision too.
Circuit breaker design for LLM fallback: comparator, gate, probe
The breaker decides when the fallback question is asked. It is not the subject of this part, and the chapter covers it in full. Under Part 3's Role Test a breaker is two components: a health comparator that judges a window of calls and holds no decision state, and a gate that refuses and remembers. resilience4j, for one, builds them as one class: its CircuitBreaker holds both the sliding window and the permission decision (resilience4j). In the same vocabulary the run budget is a bound, refusing on a count without reading the proposal. This diagram puts the three controls of this part in the order a request meets them.
direction: down
health: "health comparator\njudges the recent window, no state" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
breaker: "breaker gate\nremembers: closed, open, half-open" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
admit: "admit(fallback)\nper run, when the breaker is open" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
budget: "run budget (a bound)\nspent before every request" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
wait: "ticket waits\nor runs at propose-only" {style: {fill: "#E74C3C"; stroke: "#C0392B"; font-color: "#FFFFFF"}}
health -> breaker: "1. healthy or not"
breaker -> admit: "2. open"
breaker -> budget: "closed: primary"
admit -> budget: "3. admitted"
admit -> wait: "refused"
The comparator and the gate are fleet-wide: one breaker per dependency sees every run's calls. Admission and the budget are per run, because the floor, the band and the request count belong to one ticket.
Only a successful probe should close an open breaker. While it is open, no calls are made, so a time-based window of recent calls empties and looks healthy because nothing is being sent.
And check what your graph framework retries before you add a layer of your own. LangGraph's RetryPolicy retries nothing until you attach it. Once attached, its default retry_on retries any exception not on its exclusion list, and an httpx.HTTPStatusError only for 5xx (LangGraph source, _retry.py). Provider SDK errors such as anthropic.BadRequestError derive from Exception and are not on that list, so a 400 from the SDK is retried. That is the retry decision made from an exception type rather than a kind of failure, which the June post warned about and the chapter's config loader refuses.
Checklist: LLM fallback and retry budgets
- List what your harness sized on the primary. Repair re-asks, the run budget, refusal handling, the authority the run holds. Each one is a Plant Model measurement of the primary, even if nobody wrote it down.
- Construct the run budget once, when the run starts, and pass it down. No router, client or ladder that is built per call or per turn may own a limit that must hold per run.
- Spend the budget before every request, retries and fallback attempts included. After the response, record once more for each extra model invocation the provider reports, such as a
fallback_messageinusage.iterations. - Find out what your framework's run limit counts. Fallback rejections, counts that restart per message or after a TTL, and zero-cost models that skip budgets are all documented edges.
- Keep a wall-clock deadline beside the count. A retrying run spends most of its time waiting.
- Declare every fallback. Model id, window, the widest band it may hold, and a dated Plant Model entry measured on your current prompt template. Every model on a gateway or provider allow-list counts.
- Run the admission check when the breaker opens, before the run continues. Its floor question is the one Part 5 asks at assembly, asked earlier. Refuse a fallback whose window is below the run's floor, whose band does not cover the run, whose entry is missing, stale or measured at a narrower band, whose refusals arrive hedged above propose-only, whose measured pass rate needs more re-asks than its ladder allows, or whose worst case exceeds what is left of the run budget.
- Treat a hedged refusal as a reason for propose-only. If the fallback refuses by writing a polite non-answer, a schema check will pass it. A person reading each draft will not.
- Exempt same-weights, same-platform failover from re-measurement, and from nothing else. Its window, band and budget are still checked.
- Decide in advance what a refusal does in each mode. In queue-drain, stop picking up tickets that need the refused authority. In copilot, decide whether a propose-only run may start.
- Never lower the floor or widen a band during an incident. Measure a fallback ahead of time instead.
The gateway's dashboard answers whether the call returned. After an outage, the question that matters is what the run was still bound by when it continued on another model: which authority it held, which measurements it was sized on, and how many requests it really sent. A fallback that cannot answer those three before it takes over is a second system running your agent on the first one's authority.
Where to go deeper: LLM fallback and circuit breaking in the Harness Engineering book
This is Part 8 of a series that follows Harness Engineering for Production AI Systems one chapter at a time. Chapter 8, "Retry, Fallback, Circuit Breaking," covers what this part leaves out. It explains why a timeout on a non-idempotent write must not be retried while a rate limit may be, and why jitter matters. It builds the breaker as a comparator and a gate, with its minimum sample and its probe. It wires the retry loop into a LangGraph node with injected sleeps, and covers the stuck-open breaker you will actually have, plus the morning-after reprocess path. Code for the chapter is the ch08-retry-fallback-breaking tag of the companion repository, and its tests need no model and no API key.
Writing this part found several things for the book's next edition. At the tag, the retry loop never applies jitter although the config sets jitter = true, the graph wiring accepts a clock and never checks the wall-clock budget, and the fallback chooser reads only the first declared fallback. A typo in a fallback's max_band surfaces as a bare ValueError during an incident rather than at load. Chapter 8's "a smaller model usually means a smaller window" no longer holds for the current generation of Claude models, which list a 1M-token window (Anthropic, models overview), and its failure kinds have no slot for a refusal that arrives as a successful response.
References
- Anthropic. Configure your agent. Claude Agent SDK docs. https://code.claude.com/docs/en/agent-sdk/configuration
- Anthropic. Models overview. Claude Platform Docs. https://platform.claude.com/docs/en/about-claude/models/overview
- Anthropic. Python SDK. Claude Platform Docs. https://platform.claude.com/docs/en/api/sdks/python
- Anthropic. Refusals and fallback. Claude Platform Docs. https://platform.claude.com/docs/en/build-with-claude/refusals-and-fallback
- BerriAI. LiteLLM: Agent iteration budgets. https://docs.litellm.ai/docs/a2a_iteration_budgets
- BerriAI. LiteLLM: Budgets and rate limits. https://docs.litellm.ai/docs/proxy/users
- BerriAI. LiteLLM: Proxy reliability. https://docs.litellm.ai/docs/proxy/reliability
- BerriAI. LiteLLM: Router, fallbacks and retries. https://docs.litellm.ai/docs/routing
- Beyer, B., Jones, C., Petoff, J., & Murphy, N. R. (Eds.) (2016). Site Reliability Engineering, chapter 21, "Handling Overload". O'Reilly. https://sre.google/sre-book/handling-overload/
- Cui, J., et al. (2025). OR-Bench: An Over-Refusal Benchmark for Large Language Models. ICML 2025. https://arxiv.org/abs/2405.20947
- Envoy Project. CircuitBreakers.Thresholds.RetryBudget. https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/cluster/v3/circuit_breaker.proto
- Geng, S., et al. (2025). JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models. https://arxiv.org/abs/2501.10868
- gRPC Authors. Retry. gRPC guides. https://grpc.io/docs/guides/retry/
- Kumar, R. (2026). Harness Engineering for Production AI Systems, Chapter 8, "Retry, Fallback, Circuit Breaking". https://ranjankumar.in/harness-engineering-for-production-ai-systems
- Kumar, R. (2026). harness-engineering-code, tag
ch08-retry-fallback-breaking. https://github.com/ranjankumar-gh/harness-engineering-code/tree/ch08-retry-fallback-breaking - LangChain.
Runnable.with_fallbacks. LangChain reference. https://reference.langchain.com/python/langchain-core/runnables/base/Runnable/with_fallbacks - LangChain. LangGraph
_retry.py(default_retry_on). https://github.com/langchain-ai/langgraph/blob/main/libs/langgraph/langgraph/_internal/_retry.py - OpenAI. Running agents. OpenAI Agents SDK docs. https://openai.github.io/openai-agents-python/running_agents/
- OpenRouter. Model fallbacks. https://openrouter.ai/docs/guides/routing/model-fallbacks
- Pan, T. (2026a). The Fallback Model You Never Load-Tested. https://tianpan.co/blog/2026/05/17/fallback-model-you-never-load-tested
- Pan, T. (2026b). Retry Budgets for LLM Agents. https://tianpan.co/blog/2026-04-16-retry-budget-llm-agent-cost-amplification
- Portkey. Fallbacks. https://portkey.ai/docs/product/ai-gateway/fallbacks
- Pydantic. Pydantic AI Harness: Spend. https://pydantic.dev/docs/ai/harness/spend/
- Resilience4j. CircuitBreaker. https://resilience4j.readme.io/docs/circuitbreaker
- Wang, J. (2025). Your LLM provider will go down, but you don't have to. Assembled. https://www.assembled.com/blog/your-llm-provider-will-go-down-but-you-dont-have-to
Related Articles
- Retry, Fallback, and Circuit Breaking: Building LLM Infrastructure That Survives Outages
- Harness Engineering: The Missing Layer Between LLMs and Production Systems
- Normalization and Input Defense: Hardening the Entry Point of Your LLM System
- LLM Context Window Management: The Policy Nobody Wrote






