← Back to Guides

Companion to the book Harness Engineering for Production AI Systems · this part maps to Book Ch 5

GuideFor: AI Engineers, ML Engineers, Platform Engineers, AI Systems Architects

LLM Context Window Management: The Policy Nobody Wrote

Every fit-to-window loop runs a policy nobody wrote, and its quietest failure is a constraint summarised into a sentence that constrains nothing. Give the context budget a floor, and refuse below it before the model is called.

#harness-engineering#context-engineering#context-window#llm-reliability#langgraph#production-ai-systems

At turn thirty of a long billing ticket, the refund ceiling became two words. The window had stopped fitting, and context window management in this harness was one loop: take the oldest material and summarise it, because old material is the least likely to matter and summarising feels safer than deleting. In this harness the system prompt sits in the message list, so it is the oldest material of all. After the next pass, the window opened like this:

text
fit_to_window: 27474 tokens of 28000    system_prompt          -> 'You are the billing support assistant.'    operating_constraints  -> 'Operating constraints.'    '800.00' anywhere in the window: False

The billing ticket is constructed, from the support agent this series and its book are built around, and the summariser is the cheapest one there is: keep the first sentence. Before that pass, the ceiling of $800.00 was a paragraph. Now two words describe it. Nothing raised an exception, nothing was logged, and the function returned a context that fit, which is the only thing it was asked to do. Two turns later nothing in the window says what the model may not propose. The refund itself is still gated: as Part 3 insisted, a number that must hold lives in code, and this harness's code gate still stops any refund over the ceiling. What the window lost is everything that gate does not see: the escalation rule, the limit on what the reply may promise the customer, and the proposal quality the gate depends on. A reply that promises $1,200 and a gate that then refuses it is still a failed ticket.

Framework code shows the same shape. When LangChain's SummarizationMiddleware finds nothing left to summarise after its own trim, the source returns a fixed string, "Previous conversation was too long to summarize.", and that string stands in for the older part of the history (LangChain source, master on 2026-11-13). A user of LangChain's DeepAgents reported a related failure in October 2026, where the summariser model itself wrote an empty summary that "was then surfaced as the assistant response, and the task never completed" (LangChain forum). In the issue it links, this is called a content-free compaction that "looks successful but preserves no task state or decisions" (deepagents #6470). A better summariser would keep the rule more often, and a lead-sentence one makes the loss certain. What the demonstration shows does not depend on that choice: when an always-fits loop loses a rule, through a weak summary or an empty one, nothing in its output says so.

My claim is that an LLM context trimmer which always returns a context is running a policy nobody wrote, and that the fix is not a better summariser. It is a declared context budget with a floor, which I call the Context Floor: the spans without which the run must not happen, checked whole and before anything is evicted, so that below the floor assembly refuses before the model is called. A lookup that returns nothing, a tokeniser change or a fallback model then stops the run loudly instead of letting it continue on less. Position belongs in the same budget, as what I call Position Debt: the adherence a constraint loses because of where you placed it, measured against the best placement on your own model. It is a dated measurement, and it expires when the model changes. Under today's mixed evidence, the placement rule that holds up is a rule about conflicts: put the constraints at both ends, and never give the last slot to the customer.

Some tools already declare which spans must survive compaction. Refusing below them, before the call and against the model that call will use, is the part I have not found shipped anywhere.

This argues against advice I gave on this blog in April. That post said to "summarize everything older into a structured summary message" and, when a context runs over budget, that "compression runs again more aggressively before the LLM call proceeds". It also coined Context Gravity for the pull position exerts on attention. The name still fits the pull, but that post's claim that middle content "often doesn't meaningfully influence the output" does not survive the 2025 evidence below, and Position Debt is how you find out what holds for your model. And it builds on four earlier parts. Part 1 listed this exact production case among the ones a demo never shows: a thread so long that the refund ceiling fell out of the summarised context at turn 30. Part 2 named the Flat Context: the model receives one token sequence, so every guarantee about origin lives in code. It also measured the untrusted fraction of a turn and logged it as a per-turn signal, and the obvious next step is to cap it. Part 3 sorted components with the Role Test, which files context assembly as a sensor. Part 4 named Laundered Provenance, which is what a summary does to the text it replaces.

Why LLM context trimming that always fits hides its worst failure

Part 3 already named this failure in passing: "the most damaging sensor failure, a compactor that drops a rule from the window, is a shaping failure". This part is about that compactor.

Ask a team for its context budget and you usually get a number: 32,000 tokens, or a million. That is a size limit, and it says when to stop and nothing about what to stop doing. Your real budget is a loop somewhere in the code, usually inside a helper with a name like fit_to_window. Here is the one that produced the opening output.

python
count = PRIMARY.count_tokens  # the 4-characters-per-token proxy used throughoutdef lead_sentence(text: str) -> str:    """A cheap extractive summary: keep the first sentence."""    return text.replace("\n", " ").split(". ")[0].rstrip(". ") + "."def fit_to_window(messages: list[tuple[str, str]], limit: int) -> list[tuple[str, str]]:    """Oldest first, summarise until it fits. Always returns something."""    out = list(messages)    for i, (label, text) in enumerate(out):        if sum(count(t) for _, t in out) <= limit:            break        out[i] = (label, lead_sentence(text))    return out

Its input is the same turn-thirty spans the budgeted assembler gets later, oldest first: system prompt, constraints, account summary, invoice rows, knowledge-base articles, similar tickets, the thread's turns, and the ticket itself. It is the worst case on purpose. Most harnesses pin the system prompt: Anthropic takes it as a separate system parameter, and Claude Code keeps it through compaction. The constraints that get summarised in practice arrive as messages: a rule injected by a hook, a policy stated mid-thread, a per-account limit in a tool result. Claude Code's own table says "Context that hooks added earlier" is "Summarized with the rest of the conversation" (Claude Code docs). Three properties make this loop fail the same way every time. Eviction order is chronology used as importance, so whatever arrived first goes first. Each summary reads like the thing it replaced, so a reviewer who reads the window sees a sentence about constraints and moves on. And the function returns a list either way, so the run cannot tell a window that lost nothing from one that lost its ceiling.

In the next diagram, the left column is what the operator wrote and the right column is what the model read at turn thirty.

d2
grid-rows: 1
grid-gap: 40
written: "What the operator wrote" {
  label.near: top-center
  grid-columns: 1
  grid-gap: 12
  style: {fill: "#F4F6F7"; stroke: "#95A5A6"; font-color: "#2C2C2A"}
  s1: "system_prompt\n12 paragraphs of instructions" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  s2: "operating_constraints\nonly the duplicated charge\nnever more than 800.00\nescalate anything else" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
}
read: "What the model read at turn 30" {
  label.near: top-center
  grid-columns: 1
  grid-gap: 12
  style: {fill: "#F4F6F7"; stroke: "#95A5A6"; font-color: "#2C2C2A"}
  r1: "'You are the billing\nsupport assistant.'" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
  r2: "'Operating constraints.'\nno ceiling, no escalation rule" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
}

What makes the right-hand column dangerous is that it is not empty. An empty column would be noticed. A column with the right headings and none of the content passes a glance, a code review and a dashboard that counts tokens.

What the framework trimmer does with your system prompt

A framework trimmer is the obvious replacement for a hand-written loop, so I ran LangChain's trim_messages (langchain-core 1.8.0) on a 61-message thread whose first message is the system prompt.

python
from langchain_core.messages import trim_messages# history: a 61-message thread whose first message is the system prompttrim_messages(history, max_tokens=400, token_counter="approximate")trim_messages(history, max_tokens=400, token_counter="approximate", include_system=True)trim_messages(history, max_tokens=20, token_counter="approximate", include_system=True)
text
defaults (strategy='last')        21 of 61 msgs  397 tok (budget 400)  system: Falseinclude_system=True               20 of 61 msgs  399 tok (budget 400)  system: Trueinclude_system=True, tiny budget   1 of 61 msgs   40 tok (budget  20)  system: True

With defaults, the system prompt is the first thing to go. Its documentation says so: include_system keeps "the SystemMessage if there is one at index 0", and it defaults to False (LangChain reference). With include_system=True it is kept. With a 20-token budget, the trimmer returned the system message alone, which counts 40 tokens, so it handed back a context twice the size it was asked for, and it did not raise. That is consistent with its contract: a trimmer's job is to return messages, and refusing would look like a failure.

None of this is a bug in LangChain. trim_messages does exactly what its signature says. My point is narrower. What gets dropped first is a statement about your domain, and a framework cannot know that the refund ceiling matters more than the ninth exchange of "still waiting". If you take its default, you take a policy that nobody on your team wrote.

d2
direction: down
loop: "trimmer: drop or summarise\nuntil it fits" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
out: "returns a context\n(always)" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
model: "model call" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
lost: "what was dropped:\nnot recorded" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
loop -> out: "1. fits, or the minimum it can return"
out -> model: "2. proceeds"
loop -> lost: "side effect"

Does the API truncate long prompts? Where refusal happens today

One layer down, the refusal I am asking for already exists, which is worth knowing before you build it. Anthropic's API returns a 400 invalid_request_error ("prompt is too long") when the input alone exceeds the window, on every model (Anthropic, context windows). So Anthropic will not quietly cut your prompt. OpenAI's Responses API also fails an oversized request with a 400 by default, but its truncation: "auto" setting, now marked deprecated, drops "items from the beginning of the conversation" instead (OpenAI, Responses API reference). Always-fits behaviour comes from that kind of setting and from the trimming layer: trimmers, summarisation middleware and compaction. They can fail when something mechanical breaks, a summary call that errors or a compaction that returns nothing. None of them refuses because a span that matters is missing.

Some of that layer has moved towards declaring what must survive. Claude Code documents what survives compaction in a table: the system prompt and output style "Both still apply", and the project-root CLAUDE.md is "Re-injected from disk" (Claude Code docs). That is effectively a required tier, though nothing in that table stops the session. Anthropic's own guidance on context engineering says compaction "typically serves as the first lever in context engineering to drive better long-term coherence", and warns that "overly aggressive compaction can result in the loss of subtle but critical context whose importance only becomes apparent later" (Anthropic, 2025). I agree with the warning and part ways on the remedy. Vendor guidance answers lost context with better compaction, so the run continues. For the spans you have declared critical, the floor answers it with a refusal, so the run stops.

d2
grid-columns: 1
grid-gap: 14
yours: "Your assembler: declared budget + floor\nrefuses when a required span is absent, over its cap, or does not fit" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
fw: "Trimming layer: framework or opt-in compaction\nnever refuses on meaning" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
prov: "Provider API\nrefuses only when the input alone overflows the window" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}

Your provider refuses on size, which it can see. Only the top layer can refuse on meaning, because only your code knows which span is the refund ceiling.

Required context spans: refuse before the model call (the Context Floor)

Some spans are not trade-offs. If the operating constraints are not in the window, the run is a different system with the same name. If the account summary is missing because a lookup timed out, the model is about to reason about an account it cannot see.

By the Context Floor I mean the set of spans a run cannot go ahead without. When one of them is missing or will not fit, the harness does not degrade. It refuses, and it does so while assembling, so the model is never called. A budget with no floor will always hand you something, and that something takes the place of the refusal you needed.

Part 3's Role Test files assembly as a sensor, and a sensor cannot refuse a proposal. Nothing in the floor breaks that rule, because at assembly time there is no proposal yet. What the floor refuses is to produce a context at all, which is a different act with a different owner: the run stops, and whoever owns the cause gets the alert. Part 3's Role Test was written for actions, so placing the floor check takes one stated extension. Treat the model call as the action it guards. The check can then make that call fail every time it decides to, and it reads only the assembled inputs and the model profile, never a model proposal. Under question 3 that makes it a bound, the same role as Part 1's count ceiling, MAX_REFUNDS_PER_RUN. Around it, the assembler stays a sensor.

Each of the check's three outcomes has a different fix.

python
class FloorBreached(ValueError):  # fallback: the default retry_on skips ValueError    """What must be present is absent, over its cap, or does not fit. Nothing is sent."""    def __init__(self, kind: str, labels: tuple[str, ...], detail: str) -> None:        super().__init__(f"{kind}: {', '.join(labels)} ({detail})")        self.kind, self.labels, self.detail = kind, labels, detaildef check_floor(budget: ContextBudget, model: ModelProfile,                sized: dict[str, tuple[SpanBudget, str, int]]) -> None:    required = [s for s in budget.spans if s.requirement is Requirement.REQUIRED]    absent = tuple(s.label for s in required          # missing, or present but empty                   if s.label not in sized or sized[s.label][2] == 0)    if absent:        ours = any(budget.spec(x).origin is Origin.OPERATOR for x in absent)        why = ("an operator span was not supplied" if ours               else "an upstream lookup returned nothing")        raise FloorBreached("absent", absent, why)    over = tuple(s.label for s in required if sized[s.label][2] > s.max_tokens)    if over:        sizes = ", ".join(f"{sized[x][2]} > {sized[x][0].max_tokens}" for x in over)        raise FloorBreached("over-cap", over,                            f"{sizes} tokens on {model.name}; never truncated")    needed = sum(sized[s.label][2] * len(s.placement) for s in required)    if needed > budget.available(model):        detail = f"{needed} needed, {budget.available(model)} available on {model.name}"        raise FloorBreached("does-not-fit", tuple(s.label for s in required), detail)

My own budget missed the middle check. A required span that has grown past its own cap must not be cut to fit. Cutting the end off a constraint paragraph is the summary failure in another form: the window still opens with the right heading, and the last rule is gone. I found this in my own book's companion code. Its assembler truncates every span to its cap before it checks the floor, required spans included. When I grew the system prompt to 1,117 tokens against its 900-token cap, the assembler cut 217 tokens, the rule in its last line disappeared, and the run went ahead with an eviction record that said required:

text
system_prompt  required  -217  over its own caplast rule still in the window: False

A release that adds a paragraph to the prompt does the same thing. According to the chapter, a required span in the eviction list is impossible by construction, and the code made it routine. This version never truncates a required span. Over its cap is a refusal.

Here are the three checks in the order they run, each with its fix.

d2
grid-rows: 4
grid-columns: 2
grid-gap: 22
q1: "1. Every required span present?" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
a1: "No: absent\nfix upstream (a lookup failed)" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
q2: "2. Each one within its own cap?" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
a2: "No: over-cap\nnever truncate; a human decides" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
q3: "3. All of them fit in the\nmodel's available tokens?" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
a3: "No: does-not-fit\nthe budget no longer matches the model" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
q4: "Fill preferred, then optional" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
b4: "" {style: {opacity: 0}}
q1 -> q2
q2 -> q3
q3 -> q4

Order matters here. Because the floor is judged before the fill evicts anything, a required span never competes with a knowledge-base article for room, and the refusal names the cause rather than the symptom.

A token-counter change or a fallback model moves you below your own floor

Each check takes a ModelProfile, not just a budget, because the arithmetic belongs to the model. One file of spans can be inside the floor on one model and below it on the next, without anyone editing the file.

python
@dataclass(frozen=True)class ModelProfile:    """What the budget is spent against. Read at call time, not at load time."""    name: str    window: int    count_tokens: Callable[[str], int]

Usually the worry is a fallback to a smaller window, and in late 2026 that worry is narrower than it was. All the current Claude models (Fable 5.2, Opus 5.5, Sonnet 5.5 and Haiku 5.5) list a 1M-token window (Anthropic, context windows). What remains is a self-hosted fallback, a pinned older model and a cross-vendor fallback. Easier to miss is the tokeniser. Anthropic says 1M tokens is roughly 2.5M Unicode characters on the tokeniser introduced with Claude Opus 4.7 (Anthropic, models overview), so about 2.5 characters per token. Against that ratio, the common proxy of 4 characters per token undercounts Claude tokens by more than a third. Four was never Claude's own ratio, and a team that already uses the provider's counter sees a smaller jump on an upgrade. The arithmetic does not care which: this prompt refuses as soon as its real count passes 900, a rise of about a third over the proxy's 671. The 4.0 proxy also means this budget is illustrative: counted at Claude's ratio, the happy-path turn thirty would not fit in 28,000.

Here is turn thirty of the same ticket, run five ways: on the primary model with a 4-characters-per-token proxy, with the account lookup missing, on the same text counted at the current Claude ratio of 2.5, as a long ticket on a self-hosted fallback served with an 8,192-token limit, and with the lookup returning an empty string. The middle three are the three breach kinds. All outputs in this part are the real output of the code shown, run on Python 3.13 with langchain-core 1.8.0 and langgraph 1.2.21.

text
turn 30    24492 tokens assembled of 28000 available    kb_articles       preferred -767    cut to its own cap    customer_turns    optional  -1500   cut to its own cap    agent_turns       optional  -3600   cut to its own cap    similar_tickets   optional  -6633   retrieval ceiling: 7993 of 8400    last span: operator; constraints at head and tail: Trueturn 30, account lookup failed    REFUSED before the model call: absent    spans: account_summary    an upstream lookup returned nothingturn 30, same text, counted at Claude's ratio    REFUSED before the model call: over-cap    spans: system_prompt    1073 > 900 tokens on primary @ 2.5 chars/token; never truncatedturn 30, long ticket, fallback    REFUSED before the model call: does-not-fit    spans: system_prompt, operating_constraints, account_summary, ticket_body    4627 needed, 4192 available on self-hosted fallback, 8,192 windowturn 30, account lookup returned an empty string    REFUSED before the model call: absent    spans: account_summary    an upstream lookup returned nothing

The last block matters because a lookup that times out rarely leaves the span missing. It usually leaves an empty string. The floor treats a required span that counts zero tokens as absent. An error string such as "error: timeout", or a string of whitespace, still passes, so a production floor also takes a small validator per required span.

Nothing in the system prompt changed. It counted 671 tokens on the 4.0 proxy and 1,073 at Claude's 2.5, and the cap of 900 sat between those two numbers. A budget checked once, at load, against one counter, cannot see this. So the check has to run on every call against the model that call will actually use. That is also why the long ticket passes on the primary model (27,842 of 28,000 assembled) and refuses on the fallback.

Under the two counters, the same prompt lands on opposite sides of its cap.

d2
grid-rows: 3
grid-columns: 3
grid-gap: 16
h1: "same spans" {style: {fill: "#95A5A6"; stroke: "#6E7B7C"; font-color: "#2C2C2A"}}
h2: "model profile at call time" {style: {fill: "#95A5A6"; stroke: "#6E7B7C"; font-color: "#2C2C2A"}}
h3: "floor verdict" {style: {fill: "#95A5A6"; stroke: "#6E7B7C"; font-color: "#2C2C2A"}}
s1: "system_prompt\n2,684 characters" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
m1: "4.0 chars/token\n671 tokens, cap 900" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
v1: "inside the floor" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
s2: "system_prompt\n2,684 characters" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
m2: "2.5 chars/token\n1,073 tokens, cap 900" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
v2: "over-cap: refused" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}

Both rows have an identical left column. Only the middle column changed, and it changes in the pull request that bumps a model id, which usually does not touch the budget file.

These examples use a proxy counter so the code runs without an API key. In production, pass your provider's token counter into the profile, and keep the budget file in the same pull request as the model change it was written for.

Context budget tiers: what to declare for every span

Everything above the floor is a trade-off, and the budget declares it per span, in three tiers rather than a numeric priority. A numeric priority invites a long argument about whether retrieval is a 7 or an 8. Three tiers ask one question: if this span is missing, does the run still mean anything?

python
class Requirement(str, Enum):    REQUIRED = "required"    # if it does not fit whole, the run does not happen    PREFERRED = "preferred"  # filled before any optional span gets room    OPTIONAL = "optional"    # gets only the room left, in declared orderR, P, O = Requirement.REQUIRED, Requirement.PREFERRED, Requirement.OPTIONALHEAD, MIDDLE, TAIL = Placement.HEAD, Placement.MIDDLE, Placement.TAILBUDGET = ContextBudget(    max_tokens=32_000,    reserve_for_output=4_000,    max_retrieved_share=0.30,    spans=(        SpanBudget("system_prompt", Origin.OPERATOR, R, 900, (HEAD,)),        SpanBudget("operating_constraints", Origin.OPERATOR, R, 250, (HEAD, TAIL)),        SpanBudget("account_summary", Origin.TOOL_RESULT, R, 600, (HEAD,)),        SpanBudget("ticket_body", Origin.CUSTOMER, R, 4_000, (TAIL,)),        SpanBudget("invoice_rows", Origin.TOOL_RESULT, P, 3_000, (HEAD,)),        SpanBudget("kb_articles", Origin.RETRIEVED, P, 8_000, (MIDDLE,)),        SpanBudget("customer_turns", Origin.CUSTOMER, O, 6_000, (MIDDLE,), keep="tail"),        SpanBudget("agent_turns", Origin.MODEL, O, 6_000, (MIDDLE,), keep="tail"),        SpanBudget("similar_tickets", Origin.RETRIEVED, O, 6_000, (MIDDLE,)),    ),)

Every span carries the same decisions: its origin (Part 2's Origin), whether the run can proceed without it, how much it may take on its own, where it sits, and, when it has to be cut, whether its start or its end survives. Before assembly spends anything, reserve_for_output comes off the top. A window is shared between input and output, and Anthropic counts both: "The output Claude generates for the turn, including its extended thinking, counts too" (Anthropic, context windows). The reserve has to be at least the max_tokens you send, and what happens when input plus output overflows depends on the provider. On Claude 4.5 and newer the request is accepted, and if generation then reaches the window limit it stops with stop_reason: "model_context_window_exceeded". A self-hosted server may reject the request or cut it instead, so check yours.

The thread's history is two spans, not one. Filing the agent's own earlier words under the customer's label would be Part 4's Laundered Provenance in the downgrade direction, so the customer's turns are CUSTOMER and the agent's are Origin.MODEL. Splitting them loses the interleaving. A real assembler would emit one span per turn, each with its own origin, under the same two budget lines.

Both halves are optional and keep their tail when cut, so within each half the oldest turns go first, and both halves go before the invoice rows. Under enough pressure either half can drop to nothing, and because the customer's half is filled first, the agent's turns go before the customer's. A per-turn assembler would interleave them and evict by age across both. In a support thread the latest message and the account state are what the reply turns on, and the early turns are usually two people confirming the problem still exists. If that is not true of your workload, the tier is wrong for you, and it is one word in a file that someone can disagree with in a review.

python
# Excerpt from ContextAssembler.assemble. truncate is shown below; Eviction and# the budget types are plain frozen dataclasses with the fields named here.for tier in Requirement:                       # required, preferred, optional    for spec in self.budget.spans:        if spec.requirement is not tier or spec.label not in sized:            continue        _, text, full = sized[spec.label]        copies = len(spec.placement)        is_retrieved = spec.origin is Origin.RETRIEVED        if tier is not Requirement.REQUIRED:   # the floor already admitted these            room = (available - spent) // copies            limits = {"cut to its own cap": spec.max_tokens}            if is_retrieved:                   # on a tie, the ceiling is named                ceiling = f"retrieval ceiling: {retrieved} of {retrieval_cap}"                limits[ceiling] = (retrieval_cap - retrieved) // copies            limits[f"no room: {spent} of {available}"] = room            reason, limit = min(limits.items(), key=lambda item: item[1])            text = truncate(text, limit, count, spec.keep) if limit > 0 else ""            lost = full - count(text)            if lost:                           # one row per label                evictions.append(Eviction(spec.label, tier, lost, reason))            if not text:                continue        kept.append((spec, text))        spent += count(text) * copies        retrieved += count(text) * copies if is_retrieved else 0

Each span outside the floor is cut, on line boundaries, to the tightest of three limits: its own cap, the room left, and, for retrieved text, the room left under the retrieval ceiling. A span with no line breaks is all or nothing, so it fits whole or is dropped. That is what makes the tiers mean what they say. Preferred spans are filled first, so an optional span only ever gets the room a preferred span left behind. The eviction row names whichever limit bound. Four more details came from the companion-code review. truncate uses the injected counter rather than a fixed four characters per token, so swapping the counter really is one argument. It cuts on line boundaries, so an invoice row is never split into a different number, and it keeps the head or the tail as the span declares. Each label gets one eviction row with the tokens it actually lost. And the retrieval ceiling skips required spans, because the floor has already admitted them, so a mandatory retrieved document, such as a refund policy fetched per account, can never be evicted by the ceiling.

python
def truncate(text: str, cap: int, count: Callable[[str], int], keep: str) -> str:    """Cut to `cap` tokens on line boundaries, keeping the head or the tail."""    lines = text.splitlines(keepends=True)    def piece(n: int) -> str:        return "".join(lines[:n] if keep == "head" else lines[len(lines) - n:])    lo, hi = 0, len(lines)    while lo < hi:               # most whole lines that fit, by the injected counter        mid = (lo + hi + 1) // 2        lo, hi = (mid, hi) if count(piece(mid)) <= cap else (lo, mid - 1)    return piece(lo)
d2
direction: down
floor: "floor check: required spans whole, within caps, fit" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
tiers: "fill in tier order" {
  label.near: top-center
  grid-rows: 1
  grid-gap: 16
  style: {fill: "#F4F6F7"; stroke: "#95A5A6"; font-color: "#2C2C2A"}
  req: "required\nnever cut, never dropped" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
  pref: "preferred\ncut to cap; filled\nbefore optional" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
  opt: "optional\nonly the room left" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
}
rec: "eviction record: label, tier, tokens lost, reason" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
floor -> tiers: "1. passes"
tiers -> rec: "2. every drop named"

Half of the design is the record at the bottom. At turn thirty, 12,500 tokens of material did not reach the window, and the run knows exactly which labels and why.

Bound retrieval, not the untrusted fraction of the window

Part 2 measured the untrusted fraction of one turn at 78% and logged it as a signal. To turn it into a control, the obvious move is a cap on untrusted text as a share of the window. It is one line, max_untrusted = 0.85, and it reads like the right thing to do. Here is what it does to the same turn thirty.

python
def assemble_with_untrusted_ceiling(system: str, max_untrusted: float) -> int:    """Required spans first, then the rest in declared order, while untrusted <= share."""    texts = {c.label: c.text for c in turn(30, system=system)}    spent = untrusted = 0    tier_order = list(Requirement)    for spec in sorted(BUDGET.spans, key=lambda s: tier_order.index(s.requirement)):        tokens = min(count(texts[spec.label]), spec.max_tokens) * len(spec.placement)        is_untrusted = spec.origin not in TRUSTED_ORIGINS        share = (untrusted + tokens * is_untrusted) / (spent + tokens)        if spec.requirement is not Requirement.REQUIRED and share > max_untrusted:            continue        spent += tokens        untrusted += tokens * is_untrusted    return spent
text
untrusted <= 85%, full system prompt        : 4097 tokens of 28000untrusted <= 85%, system prompt cut by half : 941 tokens of 28000

Assembly collapsed to 4,097 tokens out of 28,000, and the arithmetic is simple once you see it. Trusted text is a small fixed quantity: here, the system prompt and two copies of the constraints, 789 tokens. If untrusted text may be at most 85% of the total, trusted text must be at least 15%, so the whole window is capped at 789 / 0.15, about 5,260 tokens.

Of that, the required spans took 1,277. Next came the invoice rows (2,820 tokens), which still fit, at an untrusted share of 81%. That left about 1,160 tokens under the cap, and every remaining span was larger than that. Halve the system prompt and the cap falls to about 3,000, so even the invoice rows push the share past 85%, and the assembly is the required spans alone: 941 tokens. Write a shorter prompt, lose most of your context. Nobody intends that relationship, and the control has no bug to find. The quantity it caps is the wrong one.

Compare the two ceilings on the same 28,000-token budget.

d2
grid-rows: 3
grid-columns: 2
grid-gap: 14
l1: "available budget" {style: {fill: "#95A5A6"; stroke: "#6E7B7C"; font-color: "#2C2C2A"}}
v1: "28,000 tokens" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
l2: "untrusted share <= 85%" {style: {fill: "#95A5A6"; stroke: "#6E7B7C"; font-color: "#2C2C2A"}}
v2: "4,097 tokens\ncapped near 789 / 0.15" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
l3: "retrieval share <= 30%" {style: {fill: "#95A5A6"; stroke: "#6E7B7C"; font-color: "#2C2C2A"}}
v3: "24,492 tokens\nretrieval held under 8,400" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}

Bound retrieval instead, as the bottom row does. A customer's ticket is as long as the customer made it, and the invoice rows are as many as the account has. Of the untrusted inputs, retrieval is the one whose size you most directly choose, and the one most willing to grow, so a ceiling on it is a decision you can actually make. In the fill above, max_retrieved_share = 0.30 gives a retrieval cap of 8,400 tokens. Knowledge-base articles take 7,993 of that, so the similar-tickets retrieval is limited to the 407 tokens left under the ceiling and records why, retrieval ceiling: 7993 of 8400, and the window still assembles to 24,492 tokens.

Because each eviction names the limit that bound, it says which problem you have. A retrieval-ceiling eviction points at your retriever's configuration. A no-room eviction points at your budget. Thirty percent is this system's number, because the billing agent's decisions turn on account state and the articles are supporting material. A documentation assistant whose answers are the retrieved text would set it much higher, and would be right to.

Lost in the middle in 2026: where to place constraints (Position Debt)

Position Debt is the adherence a constraint loses because of where you placed it: the rate at which the model keeps the constraint at its best placement, minus the rate at the placement your budget actually uses, on one model id, at one window fill, on one date. How reliably the model acts on a span depends on where the span sits, so where it goes is a budget decision that someone writes down. Whatever gap remains at the best placement is a different entry: the model's adherence to the rule at all, which is a Plant Model measurement with no position in it.

In the April post, Context Gravity names the pull that position exerts on attention, which is a property of the model. Position Debt is a number about your own system, with a date on it, and it changes when the model does. That makes it the same kind of object as Part 2's Plant Model: an entry you measure against your own inputs and expire when the model, its serving stack or your prompt template changes. "Context debt", the name Emergent Mind's topic page uses for unnecessary or duplicate tokens piling up in a window, is a different idea (Emergent Mind).

Evidence for where that gap is largest has moved since 2023, so here is what it says now:

  • Liu and colleagues found performance "is often highest when relevant information occurs at the beginning or end of the input context" (Liu et al., 2024). That was retrieval of facts from documents, on models of its period, not adherence to instructions.
  • Chroma's 2025 Context Rot study tested 18 models and found "no notable variation in performance" across 11 needle positions on its needle-in-a-haystack task, while performance "consistently degrades with increasing input length" (Chroma, 2025). Length mattered there. Position did not.
  • Veseli and colleagues report that the middle trough is strongest when inputs fill up to half the window, and that past that point "the primacy bias weakens, while recency bias remains relatively stable" (Veseli et al., 2025).
  • OpenAI's GPT-4.1 prompting guide says "If there are conflicting instructions, GPT-4.1 tends to follow the one closer to the end of the prompt", and recommends placing instructions "at both the beginning and end of the provided context" (OpenAI, 2025).
  • Anthropic's guidance points the other way for answer quality: put long documents near the top, because "Queries at the end can improve response quality by up to 30 percent in tests" (Anthropic, prompting best practices).
  • Eliav's 2026 study of instruction adherence moved instructions between the system prompt and the user turn, and found "Placement produces effects at least as large as format at N=160 in most models, but the direction is model-specific" (Eliav, 2026).

So I do not defend the 2023 U-curve as a fact about the model you deployed last month. One finding cuts the other way: OpenAI reports that if instructions appear only once, "above the provided context works better than below" (OpenAI, 2025). That is an argument for the head copy, and it is why the constraints here appear at both ends rather than only at the tail. On position under conflict, the only direct statement I found is vendor guidance, OpenAI's for GPT-4.1, now marked archived, and it reports a tendency rather than a published test. Veseli's recency result is consistent with it, and Eliav's warning that direction is model-specific is the reason this stays a dated measurement rather than a law. On the evidence there is, when two spans in the window disagree, the one nearer the end tends to win, and the conflict a billing harness cares about is a customer's ticket arguing with the refund ceiling. Untrusted text therefore never gets the last slot. If your model's curve turns out to be flat, the rule costs little: the customer's text moves one slot from the end.

python
def place(kept: list[tuple[SpanBudget, str]]) -> Context:    context = Context()    for region in Placement:        group = [(s, t) for s, t in kept if region in s.placement]        if region is Placement.TAIL:            # On conflict the later instruction tends to win: never end on untrusted text.            group.sort(key=lambda pair: pair[0].origin in TRUSTED_ORIGINS)        for spec, text in group:            context = context.add(spec.origin, text)    return context

Declared at (HEAD, TAIL), the constraints appear twice, as OpenAI's guide recommends, and the floor counts them twice. Sent through Part 2's render(), that tail copy becomes a trailing system message, which is why the mechanism below matters. Two spans share the tail: the customer's ticket and the constraints. Sorting puts trusted origins last within that group, so the ticket sits directly before the final copy of the rules. That also honours most of Anthropic's advice: the customer's question is near the end, just not at the very end. Anthropic now offers a direct mechanism for the same move. On its current models, a system message placed mid-conversation "is still applied as a system instruction rather than as ordinary user text" (Anthropic, mid-conversation system messages), so operator text can sit late in the window with its role intact. Membership in Part 2's TRUSTED_ORIGINS is the sort key, the same test behind Span.trusted. Provenance was carried so later code could ask where text came from. Here it also decides where text goes. Every run that assembles prints last span: operator, and the turn-thirty window comes out in this order:

text
turn 30 window, in order:    1 operator   2 operator   3 tool-result   4 tool-result    5 retrieved   6 customer   7 model   8 retrieved   9 customer   10 operator'800.00' at positions: [2, 10] of 10

Set that beside the opening. On the same turn-thirty input, fit_to_window left no copy of the ceiling, while the budgeted assembler kept the constraints whole, at positions 2 and 10 of 10.

d2
grid-rows: 1
grid-gap: 14
head: "HEAD" {
  label.near: top-center
  grid-columns: 1
  grid-gap: 8
  style: {fill: "#F4F6F7"; stroke: "#95A5A6"; font-color: "#2C2C2A"}
  h1: "system_prompt" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  h2: "constraints (copy 1)" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
  h3: "account, invoices" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
}
mid: "MIDDLE" {
  label.near: top-center
  grid-columns: 1
  grid-gap: 8
  style: {fill: "#F4F6F7"; stroke: "#95A5A6"; font-color: "#2C2C2A"}
  m1: "kb_articles" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
  m2: "customer and agent turns" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
  m3: "similar_tickets (under 407)" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
}
tail: "TAIL" {
  label.near: top-center
  grid-columns: 1
  grid-gap: 8
  style: {fill: "#F4F6F7"; stroke: "#95A5A6"; font-color: "#2C2C2A"}
  t1: "ticket_body (customer)" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
  t2: "constraints (copy 2)\nthe last slot" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
}

Copies of the rules open and close the window, the material the reply depends on sits next to the opening copy, and the customer's text is never the last thing the model reads.

How large your Position Debt is, I cannot tell you, and neither can a paper. Put a distinctive, checkable constraint at the head, the middle, the tail and both ends, with tickets that ask for more than the constraint allows, and count how often your model keeps the constraint at each placement. That is an afternoon of work.

d2
grid-rows: 2
grid-columns: 4
grid-gap: 14
p1: "head" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
p2: "middle" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
p3: "tail" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
p4: "both ends\n(the budget's choice)" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
r1: "kept in k of N trials" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
r2: "kept in k of N trials" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
r3: "kept in k of N trials" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
r4: "kept in k of N trials" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}

Each placement in the top row gets the same N tickets, and the bottom row is what you count. Position Debt is the best bottom-row rate minus the rate under the placement your budget uses. Record it with a unit and a date, so it can expire:

python
@dataclass(frozen=True)class PositionDebt:    model: str                     # the exact model id the trials ran against    measured_on: date              # expires when the model, stack or template changes    constraint: str                # e.g. "never propose more than 800.00 on one ticket"    window_fill: float             # share of the window in use; the curve moves with it    trials: int                    # tickets per placement, each asking for too much    held: tuple[tuple[str, int], ...]  # (placement, trials where it was kept)    used: str = "both"             # the placement your budget actually declares    @property    def debt_points(self) -> float:        """Adherence lost to placement: best placement minus the one you use."""        rate = {where: 100 * kept / self.trials for where, kept in self.held}        return max(rate.values()) - rate[self.used]

Its unit is percentage points of constraint adherence. The best placement is whichever of the four held most often, and a record that never measured the placement it claims to use raises a KeyError, which is intended. A Position Debt of zero means your placement is already the best one for that model, that fill and that constraint. It says nothing about how often the model breaks the rule anyway; that is the separate Plant Model entry. Read it against sampling noise. With 50 trials per placement a 10-point gap is inside the noise, and taking the best of four rates biases the best one upward, so record the trial count and re-run before acting on a small number. I have not put a number here. Quoting mine to justify your placement would be Borrowed Evidence, Part 1's name for justifying a system with results measured somewhere else.

Why a compaction summary can make the model ignore your rules

A rule can be present, well placed, and still lose to a paraphrase of itself. This is not Position Debt. It is distraction, and the clearest field report I found shows it. A Claude Code user who watched rules drift after compaction noted that "CLAUDE.md is not lost. It is reloaded fresh from disk after every compaction", and proposed that "The compacted summary at position 2 often restates project rules in its own words... the model gives more weight to the summary's interpretation than the actual rules" (claude-code #48959). It is one practitioner's hypothesis. Chroma's study does supply a measurement of the mechanism in general: "Even a single distractor reduces performance relative to the baseline" (Chroma, 2025).

A summary that paraphrases a required span is a distractor for that span. So if you compact, and most long-running agents do, three rules follow from the budget you already have. A required span is never summarised, and the summariser is told not to restate it. Each summary is a new span with Origin.MODEL, capped like any other. And compaction produces an eviction record, because a compactor that does not report what it compressed is the function this part opened with.

Part 4's Laundered Provenance returns here, in the role a summary enters under. LangChain's middleware inserts its summary as a user message, so model text arrives under the user's label. My April post did worse: its summary went in as {"role": "system", ...}, which promotes model-written text to operator authority. That is the upgrade direction of Laundered Provenance, the same direction as what Dabirsiaghi named compaction laundering, without the attacker (Dabirsiaghi, 2026), and the floor cannot catch it, because the floor checks which spans are present, not which role they were given.

d2
direction: down
model: "model output\n(the summary)" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
wrong: "inserted as role: system\nmodel text gains operator authority" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
right: "new span, Origin.MODEL\ncapped, recorded, never a required span" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
model -> wrong: "April post"
model -> right: "this part"

Wiring the floor into LangGraph without retrying it

In a LangGraph graph the assembler is a node, and a floor breach should leave the graph as an exception. The assembler node must not catch it: it has no information to recover a missing account summary with. A graph may route a refused run to a hand-off node that queues it for a human, as long as that route never reaches the model.

There is a trap in "it propagates". LangGraph retries nothing until you attach a RetryPolicy, but once attached, its default retry_on "retries on any exception except" a list that includes ValueError and TypeError (LangGraph fault tolerance), and you can set that policy for every node at once. A floor breach is deterministic, so retrying it repeats the same refusal and waits between attempts. I ran a node that raises the breach three ways on langgraph 1.2.21: an Exception-derived breach under RetryPolicy() with its defaults, the same breach under a policy whose retry_on excludes it by name, and the ValueError-derived FloorBreached shown earlier under the defaults.

python
library_default = RetryPolicy().retry_ondef never_retry_floor(exc: Exception) -> bool:    """Exclude floor breaches by name, then defer to the library default."""    floor = (FloorBreached, FloorBreachedAsException)    return not isinstance(exc, floor) and library_default(exc)
text
Exception subclass, default policy     3 attempt(s), then raisedException subclass, explicit retry_on  1 attempt(s), then raisedValueError subclass, default policy    1 attempt(s), then raised

FloorBreachedAsException is the probe's Exception-derived copy of the breach, which is why the tuple names both; in your own code the tuple is just (FloorBreached,). The explicit retry_on is the pattern to use. It names the breach, so it does not depend on an exclusion list inside a library that can change. The ValueError base is only a fallback for graphs whose policy you do not control, and it has a cost of its own: except ValueError: is common in parsing wrappers and retry helpers, and any of them near the assembler will catch a floor breach as ordinary bad input. Pin the behaviour with a test that attaches a retry policy, because the default graph proves nothing about the graph someone configures next quarter. LangGraph 1.2 also added node-level error handlers, which run after retries are exhausted, so a handler that returns state instead of re-raising turns the refusal into an ordinary step. Keep floor breaches out of any handler.

d2
direction: down
node: "assemble node\nraises FloorBreached" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
retry: "RetryPolicy attached:\ndefault retry_on retries any\nException except ValueError, ..." {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
three: "Exception subclass:\n3 attempts, same refusal" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
one: "explicit retry_on, or\nValueError subclass:\n1 attempt, then raised" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
node -> retry
retry -> three
retry -> one

In a graph that sets a retry policy on every node, that one decision is the difference between an on-call page now and the same page several backoff intervals later.

Operating the floor: refusal rates, evictions per tier, and the night it breaches

A floor is built to stop runs, so the first thing it does in production is stop runs, on a night when someone is on call. Its message names the breach kind, and the kind tells them which case they have.

Breach kindUsual causeRight response
absentan upstream lookup timed out or returned nothingusually nothing in the harness: the refused runs would have been wrong; fix upstream
over-cap, on an operator spana release grew a required span, or the token counter changeda human decides: raise the cap and take room from another span, or shorten the text
over-cap, on ticket_bodythe customer pasted a long log or threadroute the ticket to a human queue; never raise the cap for one request
does-not-fita fallback or a model change shrank the available tokensa human decides; check which model the failing calls used

Do not make a breach go away by shrinking reserve_for_output below the max_tokens you send. Assembly will pass, and on the window-limited fallback the cost arrives later, as replies that stop early or requests the server rejects, on the longest tickets, which are the ones where a cut-off reply does the most damage.

The customer should never see a refusal as an error. A refused run hands the ticket to a person with the breach kind attached, and the customer gets the reply the person writes. Count customer-caused refusals separately from configuration breaches: a customer can trigger the first kind on purpose, and mixed in with the second it hides the signal you alert on.

A floor's refusal rate tells you whether it is set wrong or doing its job. A floor that breaches occasionally, on unusual tickets, is working. A floor that breaches on a tenth of traffic is describing a system you no longer run. Refusal also has a cost. Some refused runs would have produced a correct reply from a degraded window, and you should count those in shadow mode before you turn the floor on.

Watch evictions per tier, recorded as one control signal per assembly in a harness-owned decision log. Optional evictions are normal. Preferred evictions every day mean a cap is wrong. A required span in an eviction list is now impossible by construction, and if one ever appears the bug is in the assembler. What should worry you is a drop to zero. An assembler that suddenly evicts nothing has usually stopped receiving something.

What would change this: when the Context Floor and Position Debt stop mattering

If position effects disappear for the model you run, measured on your own inputs, the placement rule becomes a harmless habit, and the duplicated constraints cost you about 60 tokens a turn for nothing. Anthropic's compaction already returns its summary in a compaction block you can read, which is half of the reporting (Anthropic, compaction). If compaction services also start refusing on declared required spans, most of the assembler in this part moves into the framework, where it belongs. The floor itself would not change. It would just be someone else's code.

At the top of this part, the trimming loop had one exit, and it led to the model. With the Context Floor comes the second exit: a refusal, raised before the call, that names which required span was absent, over its cap, or could not fit on the model this call uses. Everything else in the budget, the tiers, the retrieval ceiling and the placement of the last slot, is a decision someone can now read in a file and argue with. The policy that used to live in a trimming loop nobody wrote now has an owner: whoever approves changes to the budget file. The same owner keeps the dated Position Debt record beside it, and re-measures it when the model changes.

Checklist: audit your LLM context assembly

  1. Find the code that makes your context fit. Write down, in one sentence, what it drops first. Ask two colleagues to do the same. If the three sentences differ, the order belongs in a file.
  2. If you use trim_messages, check that include_system=True is set, and know that it can still return more than max_tokens.
  3. Mark the spans your run cannot proceed without as required. Assembly raises when one is absent or empty, and a small validator catches error strings.
  4. Never truncate or summarise a required span. Treat over-cap as a refusal.
  5. Check the floor on every call, against the token counter and window of the model that call uses.
  6. Bound retrieval, not the untrusted fraction. Start near the 0.30 share used here if retrieved text is supporting material, and tune it. Record which limit bound, and name the ceiling on a tie, so the eviction names the right cause.
  7. Put the constraints at both ends. Never give the last slot to untrusted text.
  8. Measure your Position Debt on your own model and inputs, date the number, and measure again when the model id, serving stack or prompt template changes.
  9. If you compact, insert the summary as Origin.MODEL, never as a system message, and record the compaction as an eviction.
  10. Exclude the floor exception by name in retry_on, and test it with a retry policy attached.
  11. Alert when the daily refusal rate rises above the rate you measured in shadow mode, when preferred-tier evictions become daily, and when evictions drop to zero after a baseline. Do not alert on total tokens.

Going deeper: Chapter 5 of the book

This is Part 5 of a series that follows Harness Engineering for Production AI Systems one chapter at a time. Chapter 5, "Context Assembly as Managed Infrastructure," builds the context budget as a TOML spec that is loaded and checked at startup. It wires assembly into a LangGraph node that lowers a run's authority when anything was evicted. It also explains where deduplication belongs, versioning the budget together with the model, and why the assembler is an inner component whose record an outer component reads across a run. Code for the chapter is the ch05-context-assembly tag of the companion repository. Its tests need no model and no API key. Both the truncation and counter issues described above are logged for the book's next edition.

References


Systems Design

Follow for more technical deep dives on AI/ML systems, production engineering, and building real-world applications:


Books by Ranjan Kumar

Federated AI Agents cover

Federated AI Agents

Right-Sized AI cover

Right-Sized AI

Harness Engineering for Production AI Systems cover

Harness Engineering

The 7 GenAI Architectures cover

The 7 GenAI Architectures

Building Real-World Agentic AI Systems with LangGraph cover

Building Real-World Agentic AI Systems

The ChatML Handbook cover

The ChatML Handbook

The Chat Templates Handbook cover

The Chat Templates Handbook

Comments