What you'll build: a reasoning-aware renderer that keeps prefix caching working
A reasoning model writes its thinking first and its answer second. Your agent has to decide which of the two to send back on the next request. Keep the wrong one and nothing fails: the server simply re-reads text it had already processed. Here is what the module you build prints for one tool step, once with the reasoning kept and once with it dropped:
kept seen=278 sent=296 break=Nonedropped seen=278 sent=277 break=231seenis how many tokens the model has processed by the end of request 1: the prompt plus what it wrote.sentis the length of request 2.breakis the first token where request 2 stops repeating what the model saw.Nonemeans request 2 starts with all 278 tokens, so a prefix cache can reuse them.231is the first token the model wrote: drop the reasoning, and nothing the model generated can be reused.
By the end of this intermediate tutorial you will have two things. An updated templatestudio/render.py passes thinking flags such as enable_thinking to each model's Hugging Face chat template. A new templatestudio/reasoning.py probes what a template does with the flag, splits a reply into reasoning and answer, stores the reasoning in the field the template reads, and runs the check above. The tutorial is for engineers who run a reasoning model such as Qwen3 behind an agent loop, through Transformers, vLLM or SGLang.
Here is the failure, as a composite of three public reports. A coding agent runs a long multi-step tool task against a self-hosted Qwen model. The client sends each step's reasoning back while the step is fresh. Later it strips that reasoning when it replays the history, because the reasoning is not part of the answer. From then on, every request differs from the one before it at the first stripped step, and the server processes everything after that step again. In opencode issue #19081, stripping reasoning on replay invalidated the cache around token 18,002: the client sent 36,995 tokens and the server re-processed 18,993 of them. In QwenLM/Qwen3.8 issue #131, a Qwen3.5 template rendered empty <think> blocks into history, so the prompt drifted from what the model had written. In vLLM issue #38488, the server itself dropped reasoning_content from incoming assistant messages. None of the three reports shows an error. The only symptoms are cost and latency.
Verified against Python 3.12.2, transformers==5.22.1, tokenizers==0.23.3, huggingface-hub==2.4.0, jinja2==3.1.6, and torch==2.15.0+cpu for step 12, on 2026-11-10.
This is Part 6 of a series and continues the project from Part 5, How to Pass Tools to apply_chat_template and Parse the Calls. It takes about 35 minutes. It downloads about 45 MB of new tokenizer files and needs no API key, no account and no Hugging Face token. The optional step 12 runs Qwen3-0.6B on your CPU with the PyTorch install from Part 5.
Prerequisites: transformers 5.22.1, the Part 5 project and three new tokenizers
You need the project from Parts 1 to 5: the chat-templates-part1/ folder with its virtual environment, the templatestudio/ package with messages.py, render.py and tools.py, and Part 5's families.py, rates.py and samples.py. This part adds one module, templatestudio/reasoning.py, twelve scripts and one optional script. It changes three functions in render.py, adds one field and one check to messages.py, and appends a few lines to families.py and samples.py. If you do not have the project, create the folder and the environment with Part 1's Prerequisites, add an empty templatestudio/__init__.py, and copy six files: templatestudio/messages.py from Part 2's complete code; templatestudio/render.py, templatestudio/tools.py, rates.py and samples.py from Part 5's complete code; and families.py from Part 4's Prerequisites.
Go to the project folder, activate the environment, and turn off the advisory that Transformers prints when PyTorch is not installed. Run the cd from the folder that contains chat-templates-part1/:
cd chat-templates-part1source .venv/bin/activateexport TRANSFORMERS_NO_ADVISORY_WARNINGS=1On Windows, activate with .venv\Scripts\activate in PowerShell or source .venv/Scripts/activate in Git Bash, and set the variable in PowerShell with $env:TRANSFORMERS_NO_ADVISORY_WARNINGS = "1". Every script in this part runs from the project root.
The versions are unchanged from Parts 1 to 5. Transformers 5.22.2 (2026-11-06) hardens the Jinja sandbox and, as Part 2 noted, rejects a message that is not a dict with a TypeError. Neither change affects how reasoning renders, so the series stays on one pin. The series needs Python 3.11 or later, because messages.py imports typing.NotRequired; it was verified on 3.12.2. If you are starting fresh, install the pins inside the activated environment:
pip install "transformers==5.22.1" "tokenizers==0.23.3" \ "huggingface-hub==2.4.0" "jinja2==3.1.6"In PowerShell, continue the line with a backtick instead of a backslash:
pip install "transformers==5.22.1" "tokenizers==0.23.3" ` "huggingface-hub==2.4.0" "jinja2==3.1.6"This part compares six templates. Three are new: Qwen3.5-0.8B and the two Qwen3 2507 models, Instruct and Thinking. Only their tokenizers download, never their weights.
A block whose first line ends in (add at the end) goes after the last line of that file, separated from it by two blank lines. In those blocks, and in blocks whose first line ends in (replace NAME), the first line only says where the code goes: do not paste it. Add the list to the end of families.py:
# families.py (add at the end)# Part 6: templates that handle reasoning, plus one family that has none.THINKING_MODELS: dict[str, str] = { "qwen3": "Qwen/Qwen3-0.6B", "qwen3.5": "Qwen/Qwen3.5-0.8B", "qwen3-instruct-2507": "Qwen/Qwen3-4B-Instruct-2507", "qwen3-thinking-2507": "Qwen/Qwen3-4B-Thinking-2507", "gemma4": GEMMA4, "llama3": FAMILIES["llama3"],}Create fetch_thinking.py, which downloads each tokenizer once and prints the length of its template:
# fetch_thinking.pyfrom families import THINKING_MODELSfrom templatestudio.render import load_tokenizerfor model_id in THINKING_MODELS.values(): template = load_tokenizer(model_id).chat_template print(f"{model_id:38} template: {len(template)} chars")Check the install and the tokenizers before you go on:
python -c "import transformers as t, jinja2 as j; print(t.__version__, j.__version__)"python fetch_thinking.pyExpected output:
5.22.1 3.1.6Qwen/Qwen3-0.6B template: 4168 charsQwen/Qwen3.5-0.8B template: 7755 charsQwen/Qwen3-4B-Instruct-2507 template: 2630 charsQwen/Qwen3-4B-Thinking-2507 template: 4049 charsgoogle/gemma-4-E2B-it template: 18567 charsNousResearch/Meta-Llama-3-8B-Instruct template: 389 charsThe first run can print a warning starting Warning: You are sending unauthenticated requests to the HF Hub. and, on Windows, one starting UserWarning: `huggingface_hub` cache-system uses symlinks by default. Both are harmless.
How reasoning moves through a chat template: two renders, one boundary
A reasoning model's reply has two parts. Qwen3 writes <think>, its reasoning, </think>, and then the answer or a tool call. Your code splits the two. The answer goes to the user, and the reasoning goes into a separate field of the stored assistant message, reasoning_content. On the next request the template renders that message again, and it decides which turns get their reasoning back and which lose it.
Most reasoning templates follow a rule that has a name in the ecosystem: interleaved thinking. Reasoning is kept for every assistant step since the last real user query, so a tool loop keeps its whole think, call, result, think trajectory. Reasoning from before that query is dropped. Fireworks' thinking-history docs define the term this way, and contrast it with preserved thinking, which keeps reasoning across user turns as well. Anthropic and vLLM use "interleaved thinking" for a related generation behaviour: a model that thinks between tool calls. In this tutorial the term means the history rule. Qwen3's template stores the boundary in a variable, ns.last_query_index.
The second render has to match the first. A server with a prefix cache, such as vLLM's automatic prefix caching or SGLang's RadixAttention, keeps the key-value state of tokens it has processed. It reuses that state when a new request starts with the same token IDs. In a tool loop, request 2 should start with request 1's prompt followed by exactly the tokens the model wrote. Any difference ends the reuse at that token, even one newline inside the reasoning.
grid-rows: 2
grid-gap: 50
r1: "Request 1\nprompt tokens" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
gen: "Model writes\n<think>reasoning</think>\n+ answer or tool call" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
split: "split_reasoning\nreasoning | answer" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
store: "Stored message\ncontent, tool_calls,\nreasoning_content" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
tpl: "Chat template\nkeeps reasoning after\nlast_query_index" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
r2: "Request 2\nmust start with\nrequest 1 + generated" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
r1 -> gen
gen -> split
split -> store
store -> tpl
tpl -> r2
r2 -> r1: "prefix_break compares" {style.stroke-dash: 4}
Read it row by row, left to right. The model controls only the yellow box. The blue boxes are the code you write in this part: the split and the stored message. The purple box is the template, which you do not change. The dashed edge is the check you build in step 6, which compares the green request with what the model actually processed, token by token.
Step 1: Pass enable_thinking to apply_chat_template through render
Goal: make render, render_ids and safe_render forward extra template variables such as enable_thinking, and see what the flag does to Qwen3's generation prompt.
Why this step: any keyword argument you give apply_chat_template that it does not use itself becomes a variable the template can read. That is how enable_thinking reaches the template. Transformers does not define the flag; one template chose to look for that name. Part 5's render passes only tools=, so it cannot turn thinking off. The same mechanism has a cost. A template ignores a variable it never reads, and it gives no warning, so a misspelled flag changes nothing and tells you nothing.
direction: down
call: "render(tokenizer, conv,\nenable_thinking=False)" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
act: "apply_chat_template\n**template_kwargs" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
tpl: "Template variable\nenable_thinking" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
out: "Generation prompt\nassistant + empty think" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
call -> act -> tpl -> out
Open templatestudio/render.py and replace render, render_ids and safe_render with these versions. Each one gains **template_kwargs and passes it on. render_ids also gains tools=, because step 6 renders tool loops to token IDs:
# templatestudio/render.py (replace render)def render( tokenizer: PreTrainedTokenizerBase, conversation: Conversation, *, add_generation_prompt: bool = True, tools: list[dict] | None = None, **template_kwargs: object,) -> str: """Render a conversation into the model's wire format, using its own template. tools is a list of JSON tool schemas. A template that has no tools branch ignores it without an error. template_kwargs, such as enable_thinking=False, become variables the template can read; one it never reads is ignored. """ return tokenizer.apply_chat_template( conversation, tools=tools, tokenize=False, add_generation_prompt=add_generation_prompt, **template_kwargs, )# templatestudio/render.py (replace render_ids)def render_ids( tokenizer: PreTrainedTokenizerBase, conversation: Conversation, *, add_generation_prompt: bool = True, tools: list[dict] | None = None, **template_kwargs: object,) -> list[int]: """Render a conversation straight to the token IDs the model receives.""" encoded = tokenizer.apply_chat_template( conversation, tools=tools, tokenize=True, add_generation_prompt=add_generation_prompt, **template_kwargs, ) return encoded["input_ids"]# templatestudio/render.py (replace safe_render)def safe_render( tokenizer: PreTrainedTokenizerBase, conversation: Conversation, *, add_generation_prompt: bool = True, tools: list[dict] | None = None, **template_kwargs: object,) -> str: """Check the input contract, then render. Use this in production.""" assert_valid(conversation) return render( tokenizer, conversation, add_generation_prompt=add_generation_prompt, tools=tools, **template_kwargs, )load_tokenizer, visible_whitespace, first_divergence and render_continuation stay as they are. Create show_thinking_prompt.py. It prints only the open assistant turn at the end of the prompt, then renders once more with the flag misspelled:
# show_thinking_prompt.pyfrom families import THINKING_MODELSfrom rates import QUESTIONfrom templatestudio.render import load_tokenizer, rendertokenizer = load_tokenizer(THINKING_MODELS["qwen3"])for flag in (None, True, False): kwargs = {} if flag is None else {"enable_thinking": flag} prompt = render(tokenizer, QUESTION, **kwargs) turn = prompt.split("<|im_start|>")[-1] # the open assistant turn print(f"enable_thinking={str(flag):5} {turn!r}")typo = render(tokenizer, QUESTION, enable_thinkng=False) # misspelled on purposeprint("misspelled flag changed the prompt:", typo != render(tokenizer, QUESTION))Run it:
python show_thinking_prompt.pyExpected output:
enable_thinking=None 'assistant\n'enable_thinking=True 'assistant\n'enable_thinking=False 'assistant\n<think>\n\n</think>\n\n'misspelled flag changed the prompt: FalseWhat just happened: with no flag, and with enable_thinking=True, Qwen3's prompt ends at assistant\n, and the model decides to open its own <think> block. With enable_thinking=False, the template writes an empty block into the prompt. The model starts after </think> and answers directly. So thinking off is not a mode the model switches into. It is four tokens the template adds to the prompt, and step 8 counts them. The misspelled enable_thinkng=False rendered the default prompt, with no error and no warning. Every later step passes flags through render, and step 2 shows how to tell whether a template reads a flag at all.
Step 2: Check whether a chat template reads enable_thinking
Goal: add thinking_default to a new module, templatestudio/reasoning.py, and use it to find out what six templates do with enable_thinking.
Why this step: the flag means something different from one template to the next, and sometimes it means nothing. A model family can change its default between releases, or drop the flag. The Qwen3 2507 release split the family into separate Instruct and Thinking checkpoints. Their templates never read enable_thinking, so the checkpoint you load is the switch. Pass enable_thinking=False to Qwen3-4B-Thinking-2507 and you still get reasoning, with no error. The model card alone will not tell you this, but you can measure it. Render the same prompt three times, with the flag unset, on and off, and compare the strings.
direction: down
conv: "Same conversation" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
renders: "Three renders" {
grid-columns: 3
grid-gap: 20
u: "flag unset" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
on: "flag=True" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
off: "flag=False" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
}
verdict: "on == off: ignored\nunset == on: default on\nunset == off: default off" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
conv -> renders
renders -> verdict: "compare strings"
This is the same differential probing Part 4 used for system prompts: you learn what a template does from what it renders, without reading its Jinja. Create templatestudio/reasoning.py. The import block and the two constants below serve the whole module. You write them once here, and later steps use the names this step does not:
# templatestudio/reasoning.pyfrom __future__ import annotationsfrom dataclasses import dataclassfrom typing import castfrom transformers import PreTrainedTokenizerBasefrom templatestudio.messages import Conversation, Messagefrom templatestudio.render import first_divergence, render, render_idsOPEN, CLOSE = "<think>", "</think>"# The names reasoning travels under: the same names, in the same order of# preference, as Part 3's jinja_env.py.REASONING_FIELDS = ("reasoning_content", "thinking", "reasoning")def thinking_default( tokenizer: PreTrainedTokenizerBase, conversation: Conversation, flag: str = "enable_thinking",) -> str: """Report what a template does with a thinking flag. It renders the generation prompt three times: without the flag, with flag=True and with flag=False, and compares the three strings. """ unset = render(tokenizer, conversation) on = render(tokenizer, conversation, **{flag: True}) off = render(tokenizer, conversation, **{flag: False}) if on == off: return "ignored" if unset == on: return "default on" if unset == off: return "default off" return "unset differs from both"Create compare_thinking.py. The last column reports whether the default prompt already ends inside an open <think> block:
# compare_thinking.pyfrom families import THINKING_MODELSfrom rates import QUESTIONfrom templatestudio.reasoning import OPEN, thinking_defaultfrom templatestudio.render import load_tokenizer, renderprint(f"{'family':20} {'enable_thinking':16} prompt opens <think>")for family, model_id in THINKING_MODELS.items(): tokenizer = load_tokenizer(model_id) verdict = thinking_default(tokenizer, QUESTION) opens = render(tokenizer, QUESTION).rstrip("\n").endswith(OPEN) print(f"{family:20} {verdict:16} {opens}")Run it:
python compare_thinking.pyExpected output:
family enable_thinking prompt opens <think>qwen3 default on Falseqwen3.5 default off Falseqwen3-instruct-2507 ignored Falseqwen3-thinking-2507 ignored Truegemma4 default off Falsellama3 ignored FalseWhat just happened: six templates gave four different behaviours. Qwen3 thinks unless you pass False. Qwen3.5 does the opposite: it writes the empty block unless you pass True, so code that relied on Qwen3's default gets no reasoning on Qwen3.5. Gemma 4 also defaults to off, but its flag works differently. True puts a <|think|> token into the system turn at the top, not into the generation prompt. The two 2507 templates ignore the flag. The Thinking one opens <think> in the prompt itself, so its output contains only the closing </think>. Llama 3's template has no reasoning support, so the flag means nothing there. Step 3 handles both kinds of output, with the opening tag and without it.
Step 3: Split the reasoning from the answer in the model's output
Goal: add a Reply dataclass and split_reasoning, which cut raw model output into reasoning and visible answer.
Why this step: you need the two halves separately. The answer goes to the user and the tool parser, and the reasoning goes into the stored message. Three shapes of output break a naive split on </think>. A Thinking-2507 model writes no opening tag, because the prompt already wrote it. A reply cut off by max_new_tokens has an opening tag and no closing tag. With thinking off there is no block at all. A naive split returns the reasoning as the answer in the first case, and the whole output as the answer in the second. split_reasoning handles all three and records whether the block was closed.
direction: right
raw: "Raw output\n<think>\\nR\\n</think>\\n\\nA" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
split: "split_reasoning\nprompt_opened?" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
reply: "Reply" {
grid-columns: 1
grid-gap: 10
r: "reasoning: R" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
a: "answer: A" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
c: "closed: True" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
}
raw -> split -> reply
Append both to templatestudio/reasoning.py:
# templatestudio/reasoning.py (add at the end)@dataclass(frozen=True)class Reply: """One assistant turn, split into its reasoning and its visible answer.""" reasoning: str answer: str closed: bool # False when the output stopped inside the reasoning blockdef split_reasoning(text: str, *, prompt_opened: bool = False) -> Reply: """Split raw model output at its reasoning block. Set prompt_opened when the generation prompt already wrote <think>, so the output starts inside the block and only the closing tag appears. """ if prompt_opened and OPEN not in text: text = OPEN + text start = text.find(OPEN) if start == -1: return Reply("", text.strip(), closed=True) end = text.find(CLOSE, start) if end == -1: return Reply(text[start + len(OPEN) :].strip("\n"), "", closed=False) reasoning = text[start + len(OPEN) : end].strip("\n") return Reply(reasoning, text[end + len(CLOSE) :].strip(), closed=True)The reasoning is stripped of newlines at both ends because Qwen3's template does the same, with reasoning_content.strip('\n'), when it renders the reasoning back. Now add one reasoned tool step to the end of samples.py. It has the shape Qwen3 writes with thinking on, and it reuses Part 5's QWEN3_CALL:
# samples.py (add at the end)# Part 6: one reasoned tool step, as Qwen3 writes it with thinking on.QWEN3_REASONING = "The user wants EUR to JPY. I should call get_exchange_rate."QWEN3_REASONED_CALL = ( "<think>\n" + QWEN3_REASONING + "\n</think>\n\n" + QWEN3_CALL + "<|im_end|>")The sample ends with <|im_end|>, the token Qwen3 writes to close its turn. The model generates that token, and step 6 counts it. Create show_split.py:
# show_split.pyfrom samples import QWEN3_REASONED_CALLfrom templatestudio.reasoning import split_reasoningoutputs = { "closed block": (QWEN3_REASONED_CALL, False), "prompt opened it": ("Use the rate.\n</think>\n\nAbout 162 yen.", True), "cut off": ("<think>\nThe user wants EUR to", False), "thinking off": ("One euro buys about 162.4 yen.", False),}for label, (text, opened) in outputs.items(): reply = split_reasoning(text, prompt_opened=opened) print(f"{label}: closed={reply.closed}") print(f" reasoning: {reply.reasoning!r}") print(f" answer: {reply.answer[:44]!r}")Run it:
python show_split.pyExpected output:
closed block: closed=True reasoning: 'The user wants EUR to JPY. I should call get_exchange_rate.' answer: '<tool_call>\n{"name": "get_exchange_rate", "a'prompt opened it: closed=True reasoning: 'Use the rate.' answer: 'About 162 yen.'cut off: closed=False reasoning: 'The user wants EUR to' answer: ''thinking off: closed=True reasoning: '' answer: 'One euro buys about 162.4 yen.'What just happened: the closed block split into the reasoning and Part 5's tool call, which HermesParser can now read. With prompt_opened=True, output that starts mid-block split the same way. The cut-off reply came back with closed=False and an empty answer. Showing the user '' would be wrong. Showing half a thought as the answer would be worse. The flag lets your display layer choose. Output with no block came back as all answer. The script prints only the first 44 characters of each answer to keep the lines short.
Step 4: See which turns keep their reasoning with last_query_index
Goal: add last_query_index, a Python copy of the boundary Qwen3's template computes, and watch the template drop a turn's reasoning once a new user message arrives.
Why this step: before you can tell whether a render matches what the model saw, you need to know which stored reasoning the template will render. Qwen3's template walks the messages backwards. It stops at the latest user message whose content is not a tool result wrapped in <tool_response> tags. That index is ns.last_query_index. Assistant turns after it keep their reasoning, and turns before it render as the answer alone. The template skips the wrapped messages because some clients send tool results back as user messages in those tags. Without the skip, a tool loop would lose its reasoning halfway through.
grid-rows: 2
grid-gap: 50
u1: "user\nHow many yen...?" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
a1: "assistant\nreasoning dropped" {style: {fill: "#D3D3D3"; stroke: "#A0A0A0"; font-color: "#2C2C2A"}}
u2: "user\nAnd two euros?\n= last_query_index" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
a2: "assistant\ntool call\nreasoning kept" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
t2: "tool\n162.4" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
a3: "assistant\nanswer\nreasoning kept" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
u1 -> a1 -> u2 -> a2 -> t2 -> a3
The purple message is the boundary. The grey turn before it loses its reasoning, and the green turns after it keep theirs. The script below renders the first three boxes; steps 5 to 7 build the tool loop after them. Append last_query_index to templatestudio/reasoning.py:
# templatestudio/reasoning.py (add at the end)def last_query_index(conversation: Conversation) -> int: """Index of the last real user turn, the way Qwen3's template finds it. A user message that only wraps a tool result in <tool_response> tags is not a query, so the walk skips it. """ for index in range(len(conversation) - 1, -1, -1): message = conversation[index] content = message.get("content") if message["role"] != "user" or not isinstance(content, str): continue wrapped = content.startswith("<tool_response>") and content.endswith( "</tool_response>" ) if not wrapped: return index return len(conversation) - 1Create show_history_rule.py. It renders the same answer twice: once as the latest turn, and once with a new user question after it:
# show_history_rule.pyfrom families import THINKING_MODELSfrom templatestudio.reasoning import last_query_indexfrom templatestudio.render import load_tokenizer, rendertokenizer = load_tokenizer(THINKING_MODELS["qwen3"])first = {"role": "user", "content": "How many yen is one euro?"}answer = { "role": "assistant", "content": "One euro is 162.4 yen.", "reasoning_content": "Use the rate.",}second = {"role": "user", "content": "And two euros?"}for conversation in ([first, answer], [first, answer, second]): prompt = render(tokenizer, conversation, add_generation_prompt=False) turn = [t for t in prompt.split("<|im_start|>") if t.startswith("assistant")] print(f"last query at {last_query_index(conversation)}:") print(f" {turn[0]!r}")Run it:
python show_history_rule.pyExpected output:
last query at 0: 'assistant\n<think>\nUse the rate.\n</think>\n\nOne euro is 162.4 yen.<|im_end|>\n'last query at 2: 'assistant\nOne euro is 162.4 yen.<|im_end|>\n'What just happened: while the answer was the latest turn, after the query at index 0, the template rendered its reasoning back. Once the user asked a follow-up at index 2, the same message rendered without its block. You changed nothing in the stored message. Only the boundary moved. Qwen3's model card describes this rule as no thinking content in history, and the template carries it out for you. Steps 5 and 6 deal with the turns the template keeps, and step 8 with the turn it drops.
Step 5: Store reasoning in reasoning_content on the assistant message
Goal: add keep_reasoning, which puts a reply's reasoning into the reasoning_content field of a stored message.
Why this step: the template keeps reasoning only if the stored message carries it, and Qwen3's template reads it from reasoning_content. Part 5's tool_call_message builds an assistant turn with only tool_calls, so a tool loop built from it drops every step's reasoning. keep_reasoning adds the field to any message Part 5 builds. It refuses a reply that was cut off inside its block, because storing half a thought would render text the model never finished.
direction: right
call: "tool_call_message\nrole, tool_calls" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
reply: "Reply\nreasoning, closed" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
keep: "keep_reasoning" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
msg: "Stored message\n+ reasoning_content" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
call -> keep
reply -> keep
keep -> msg
Append keep_reasoning to templatestudio/reasoning.py:
# templatestudio/reasoning.py (add at the end)def keep_reasoning(message: Message, reply: Reply) -> Message: """Return message with the reply's reasoning where templates read it.""" if not reply.closed: raise ValueError("reply stopped inside its reasoning block; do not store it") return {**message, "reasoning_content": reply.reasoning}Create show_keep_reasoning.py. It splits the step 3 sample, parses the call with Part 5's HermesParser, stores the step with its reasoning, and renders the next request. In terms of Part 5's render/parse asymmetry, the split is the way-out half, and keep_reasoning hands the reasoning back to the way-in half:
# show_keep_reasoning.pyfrom families import THINKING_MODELSfrom rates import QUESTION, get_exchange_ratefrom samples import QWEN3_REASONED_CALLfrom templatestudio.reasoning import keep_reasoning, split_reasoningfrom templatestudio.render import load_tokenizer, safe_renderfrom templatestudio.tools import ( HermesParser, tool_call_message, tool_result_message, tool_schema,)tokenizer = load_tokenizer(THINKING_MODELS["qwen3"])tools = [tool_schema(get_exchange_rate)]reply = split_reasoning(QWEN3_REASONED_CALL)call = HermesParser().parse(reply.answer)[0]step = keep_reasoning(tool_call_message(call.name, call.arguments), reply)conversation = [*QUESTION, step, tool_result_message(call.name, "162.4")]prompt = safe_render(tokenizer, conversation, tools=tools)start = prompt.index("<|im_start|>assistant")print(prompt[start : prompt.index("<|im_end|>", start)])Run it:
python show_keep_reasoning.pyExpected output:
<|im_start|>assistant<think>The user wants EUR to JPY. I should call get_exchange_rate.</think><tool_call>{"name": "get_exchange_rate", "arguments": {"base": "EUR", "quote": "JPY"}}</tool_call>What just happened: the tool step rendered with its reasoning inside the <think> block. It did so because the step comes after the last user query and now carries reasoning_content. safe_render ran Part 2's checks first, and they passed: no InvalidConversation was raised. A render that looks correct can still break the cache, so step 6 checks this one against the token IDs the model wrote.
Step 6: Check that the next request repeats what the model saw
Goal: add prefix_break, which reports the first token where request 2 stops repeating request 1's prompt and the generated tokens.
Why this step: a prefix cache compares token IDs, not text that looks similar. Two renders can read the same to you and still differ by one newline token. Everything after that token is then processed again. The test is mechanical. Render request 1 to IDs, append the IDs the model generated, render request 2 to IDs, and find the first position where they differ. Part 1's first_divergence already finds that position. prefix_break cuts request 2 to the length of what the model saw, so a longer request that starts with all the seen tokens counts as a match.
direction: down
seen: "seen = render_ids(request) + generated_ids" {
grid-columns: 2
grid-gap: 0
p1: "request 1 prompt\n231 tokens" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
g: "generated\n47 tokens" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
}
sent: "sent = render_ids(next_request)" {
grid-columns: 3
grid-gap: 0
p2: "same 231 tokens" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
g2: "same 47 tokens?" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
new: "tool result +\nprompt, 18 tokens" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
}
seen -> sent: "first_divergence(seen, sent[:len(seen)])"
Append prefix_break to templatestudio/reasoning.py:
# templatestudio/reasoning.py (add at the end)def prefix_break( tokenizer: PreTrainedTokenizerBase, request: Conversation, generated_ids: list[int], next_request: Conversation, *, tools: list[dict] | None = None, **template_kwargs: object,) -> int | None: """First token where next_request stops repeating what the model saw. The model saw request's prompt and then wrote generated_ids. If the next prompt starts with exactly those tokens, a prefix cache can reuse them and this returns None. Otherwise it returns the first index that differs. """ seen = render_ids(tokenizer, request, tools=tools, **template_kwargs) seen = seen + generated_ids sent = render_ids(tokenizer, next_request, tools=tools, **template_kwargs) return first_divergence(seen, sent[: len(seen)])Create show_prefix_break.py. It rebuilds request 2 in three ways from the same generated text: with the reasoning kept, with it dropped, and for a model that wrote <think> with no newline after it:
# show_prefix_break.pyfrom families import THINKING_MODELSfrom rates import QUESTION, get_exchange_ratefrom samples import QWEN3_REASONED_CALLfrom templatestudio.reasoning import keep_reasoning, prefix_break, split_reasoningfrom templatestudio.render import load_tokenizer, render_idsfrom templatestudio.tools import ( HermesParser, tool_call_message, tool_result_message, tool_schema,)tokenizer = load_tokenizer(THINKING_MODELS["qwen3"])tools = [tool_schema(get_exchange_rate)]def check(label: str, generated: str, keep: bool) -> None: """Rebuild request 2 from generated text and report where it breaks.""" ids = tokenizer(generated, add_special_tokens=False)["input_ids"] reply = split_reasoning(generated) call = HermesParser().parse(reply.answer)[0] step = tool_call_message(call.name, call.arguments) if keep: step = keep_reasoning(step, reply) request = list(QUESTION) next_request = [*request, step, tool_result_message(call.name, "162.4")] where = prefix_break(tokenizer, request, ids, next_request, tools=tools) sent = len(render_ids(tokenizer, next_request, tools=tools)) seen = len(render_ids(tokenizer, request, tools=tools)) + len(ids) print(f"{label:18} seen={seen} sent={sent} break={where}")check("kept", QWEN3_REASONED_CALL, keep=True)check("dropped", QWEN3_REASONED_CALL, keep=False)check("no newline", QWEN3_REASONED_CALL.replace("<think>\n", "<think>"), keep=True)Run it:
python show_prefix_break.pyExpected output:
kept seen=278 sent=296 break=Nonedropped seen=278 sent=277 break=231no newline seen=277 sent=296 break=232What just happened: with the reasoning kept, request 2 is 296 tokens and starts with all 278 the model processed. Only 18 tokens are new: the tool result and the next generation prompt.
With the reasoning dropped, request 2 is 277 tokens, shorter than what the model saw, and it differs at 231, the first token the model wrote. The server cannot reuse any of the 47 generated tokens. It processes the 46 tokens from 231 on, instead of 18.
The third case is the subtle one. The model wrote <think> directly followed by its reasoning, but the template always renders <think> and then a newline, so the two disagree at token 232. Here, storing the reasoning was not enough: the template's spacing also has to match the model's. Your stored message cannot change the template's spacing, so measure it once per model: run prefix_break on IDs the model really generated, as step 12 does, and look for a break one or two tokens after the first generated token. With one short step, all three costs look small. Step 7 shows how a break grows when more of the loop comes after it.
Step 7: Measure the prefix cache miss when a client strips reasoning on replay
Goal: measure what the server re-processes when a client keeps a step's reasoning in one request and strips it from a later one.
Why this step: in step 6 the cache broke at the newest step, so only that step was processed again. A client that strips reasoning when it replays the history breaks the cache at the oldest stripped step instead. The server then processes everything after that point again, including every later step's reasoning and tool results. This is the opencode pattern from the top of the page. Its cost grows with the length of the loop.
direction: right
s1: "step 1\nreasoning stripped\nbreak here" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
s2: "step 2\nreasoning kept\nprocessed again" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
s3: "step 3\nreasoning kept\nprocessed again" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
s4: "step 4\nnew tokens\nprocessed" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
s1 -> s2 -> s3 -> s4
The orange step is where the next request first differs. The yellow steps after it were all cached in the previous request, but the server processes them again anyway, before the green step's new tokens.
Create show_replay_strip.py. It sends three tool steps back with their reasoning and lets the model write a fourth. Then it builds the next request two ways: with every step kept, and with tool step 1's reasoning stripped on replay. To keep the script short, the three steps reuse the same sample call:
# show_replay_strip.pyfrom families import THINKING_MODELSfrom rates import QUESTION, get_exchange_ratefrom samples import QWEN3_REASONED_CALLfrom templatestudio.reasoning import keep_reasoning, prefix_break, split_reasoningfrom templatestudio.render import load_tokenizer, render_idsfrom templatestudio.tools import tool_call_message, tool_result_message, tool_schematokenizer = load_tokenizer(THINKING_MODELS["qwen3"])tools = [tool_schema(get_exchange_rate)]ids = tokenizer(QWEN3_REASONED_CALL, add_special_tokens=False)["input_ids"]call = tool_call_message("get_exchange_rate", {"base": "EUR", "quote": "JPY"})kept = keep_reasoning(call, split_reasoning(QWEN3_REASONED_CALL))result = tool_result_message("get_exchange_rate", "162.4")# Three tool steps sent back with their reasoning; the model then writes step 4.request = [*QUESTION, kept, result, kept, result, kept, result]for label, first in (("kept", kept), ("tool step 1 stripped", call)): later = request[len(QUESTION) + 1 :] # every message after step 1 next_request = [*QUESTION, first, *later, kept, result] where = prefix_break(tokenizer, request, ids, next_request, tools=tools) seen = len(render_ids(tokenizer, request, tools=tools)) + len(ids) sent = len(render_ids(tokenizer, next_request, tools=tools)) reused = seen if where is None else where print(f"{label:21} break={where} processed={sent - reused} of {sent}")Run it:
python show_replay_strip.pyExpected output:
kept break=None processed=18 of 491tool step 1 stripped break=231 processed=241 of 472processed is how many tokens of the next request the server has to process: everything after the reused prefix. The count is per token. vLLM's automatic prefix caching reuses whole blocks of tokens (its design notes say "We only cache full blocks."), so a real server rounds the reused prefix down to a block boundary.
What just happened: when every step was kept, the server processed only the 18 new tokens. Stripping tool step 1 on replay moved the break back to token 231. The server processed 241 tokens: steps 2 to 4, with their reasoning and results, all over again. The longer the loop after the stripped step, the bigger that number gets. In a much longer session, the same mechanism produced the opencode report's 18,993 re-processed tokens out of 36,995. The model also lost its own reasoning from step 1, and keeping that reasoning is the reason the interleaved rule exists.
Step 8: Measure the prefix cache breaks Qwen3 and Qwen3.5 make by design
Goal: measure the breaks the templates cause on purpose, so you can tell them apart from the ones your code causes.
Why this step: on Qwen3, prefix_break never returns None across a new user message, and that is by design. The interleaved rule drops the previous answer's reasoning once a new query arrives. The cache can reuse everything up to that answer, but it must re-read the answer itself. Thinking off has its own break. The generation prompt contains an empty block, and Qwen3's template does not render that empty block back into a kept tool step. Qwen3.5's template does. Know these numbers before you set an alert on a broken cache.
direction: down
on: "New user turn, thinking on\nlast answer loses its block" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
off: "New user turn, thinking off\nempty block is not re-rendered" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
tool3: "Qwen3 tool step, thinking off\nbreak at the step" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
tool35: "Qwen3.5 tool step, thinking off\nno break" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
on -> off: "by design" {style.stroke-dash: 4}
tool3 -> tool35: "template choice" {style.stroke-dash: 4}
Create show_expected_breaks.py. It measures four cases and reports, as processed, how many tokens of request 2 the server has to process. Qwen3.5 writes tool calls in its own XML-style format, so its sample call differs from Qwen3's:
# show_expected_breaks.pyfrom transformers import PreTrainedTokenizerBasefrom families import THINKING_MODELSfrom rates import QUESTION, get_exchange_ratefrom samples import QWEN3_CALLfrom templatestudio.messages import Conversationfrom templatestudio.reasoning import prefix_breakfrom templatestudio.render import load_tokenizer, render_idsfrom templatestudio.tools import tool_call_message, tool_result_message, tool_schemaANSWER = "One euro is 162.4 yen."FOLLOW_UP = {"role": "user", "content": "And two euros?"}QWEN35_CALL = ( "<tool_call>\n<function=get_exchange_rate>\n" "<parameter=base>\nEUR\n</parameter>\n" "<parameter=quote>\nJPY\n</parameter>\n" "</function>\n</tool_call>")tools = [tool_schema(get_exchange_rate)]def report( label: str, tokenizer: PreTrainedTokenizerBase, request: Conversation, text: str, next_request: Conversation, **kwargs: object,) -> None: """Print where next_request breaks and how many of its tokens get processed.""" ids = tokenizer(text, add_special_tokens=False)["input_ids"] where = prefix_break(tokenizer, request, ids, next_request, **kwargs) seen = len(render_ids(tokenizer, request, **kwargs)) + len(ids) sent = len(render_ids(tokenizer, next_request, **kwargs)) reused = seen if where is None else where print(f"{label:30} break={where} processed={sent - reused} of {sent}")qwen3 = load_tokenizer(THINKING_MODELS["qwen3"])answer = {"role": "assistant", "content": ANSWER, "reasoning_content": "Use the rate."}report( "qwen3 new user turn, on", qwen3, list(QUESTION), "<think>\nUse the rate.\n</think>\n\n" + ANSWER + "<|im_end|>", [*QUESTION, answer, FOLLOW_UP],)report( "qwen3 new user turn, off", qwen3, list(QUESTION), ANSWER + "<|im_end|>", [*QUESTION, {"role": "assistant", "content": ANSWER}, FOLLOW_UP], enable_thinking=False,)step = tool_call_message("get_exchange_rate", {"base": "EUR", "quote": "JPY"})loop = [*QUESTION, step, tool_result_message("get_exchange_rate", "162.4")]report( "qwen3 tool step, off", qwen3, list(QUESTION), QWEN3_CALL + "<|im_end|>", loop, tools=tools, enable_thinking=False,)qwen35 = load_tokenizer(THINKING_MODELS["qwen3.5"])report( "qwen3.5 tool step, off", qwen35, list(QUESTION), QWEN35_CALL + "<|im_end|>", loop, tools=tools, enable_thinking=False,)Run it:
python show_expected_breaks.pyExpected output:
qwen3 new user turn, on break=15 processed=25 of 40qwen3 new user turn, off break=15 processed=29 of 44qwen3 tool step, off break=231 processed=50 of 281qwen3.5 tool step, off break=None processed=22 of 417What just happened: across a new user turn, both modes broke at token 15, right after <|im_start|>assistant\n, where the previous answer starts. With thinking on, the template dropped the answer's block. With thinking off, it did not render back the empty block from the generation prompt. Either way the server re-reads one short answer. That is the price of the history rule.
The Qwen3 tool step with thinking off broke at 231. With thinking off, request 1's prompt is 235 tokens: the 231 from step 6 plus the four tokens of the empty block from step 1. So the break sits four tokens before the end of the prompt, at the empty block. Qwen3's template renders a kept step's block only when its reasoning is non-empty. Qwen3.5's template renders the block on every kept step, empty or not, so the same loop matched and the server processed only the 22 new tokens. You can force the empty block back on Qwen3 by storing reasoning_content="\n". That works only because of how this one template strips whitespace, so the module does not do it. Without the break, Qwen3 would process only the 18 new tokens, the tool result and the next generation prompt. So the break costs 32 extra tokens a step. That is usually cheaper than a hack you have to recheck after every template update.
Step 9: Read reasoning stored under reasoning, thinking or reasoning_content
Goal: add normalize_reasoning, which moves reasoning found under any of the three names into reasoning_content, and declare the field in Part 2's input contract.
Why this step: the field has a different name at almost every layer of the stack. Gemma 4's own response template returns it as thinking. vLLM renamed its response field from reasoning_content to reasoning in v0.11.1 and removed the old name in v0.16.0. SGLang still returns reasoning_content. Transformers 5.22.1 copies thinking and reasoning into reasoning_content before rendering, so the template finds the reasoning either way. Part 3's alias_reasoning rebuilt that aliasing, with the same three names in the same order as REASONING_FIELDS here. Your own code still needs one name, for step 10's check and for your storage. A client that reads the wrong name gets an empty string, and no error.
direction: right
srcs: "Where reasoning comes from" {
grid-columns: 1
grid-gap: 12
g: "Gemma 4 parse_response\nthinking" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
v: "vLLM 0.16+ response\nreasoning" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
s: "SGLang response\nreasoning_content" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
}
norm: "normalize_reasoning" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
one: "Stored message\nreasoning_content only" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
srcs -> norm -> one
Append normalize_reasoning to templatestudio/reasoning.py. The cast tells a type checker that the cleaned dict is a Message:
# templatestudio/reasoning.py (add at the end)def normalize_reasoning(message: dict) -> Message: """Store reasoning found under any known name as reasoning_content. The first non-empty field in REASONING_FIELDS wins; the other names are dropped, so the message carries one copy under one name. """ found = [message[f] for f in REASONING_FIELDS if message.get(f)] clean = {k: v for k, v in message.items() if k not in REASONING_FIELDS} if found: clean["reasoning_content"] = found[0] return cast(Message, clean)Part 2's Message type does not declare reasoning_content yet. Its validator ignores keys it does not know, so nothing has failed so far. Still, a type checker would flag every message that keep_reasoning and normalize_reasoning return, and nothing checks that the field is a string. Open templatestudio/messages.py and add one line to Message:
# templatestudio/messages.py (replace Message)class Message(TypedDict): role: Role content: NotRequired[Content | None] tool_calls: NotRequired[list[ToolCall]] reasoning_content: NotRequired[str]Then replace validate_conversation. The only change from Part 2's version is the four lines that start with reasoning = message.get("reasoning_content"), a type check for the new field:
# templatestudio/messages.py (replace validate_conversation)def validate_conversation(conversation: Conversation) -> list[str]: """Return every input-contract violation. An empty list means well-formed.""" if not conversation: return ["conversation is empty"] problems: list[str] = [] for i, message in enumerate(conversation): if not isinstance(message, dict): kind = type(message).__name__ problems.append(f"message {i}: must be a dict, got {kind}") continue role = message.get("role") if role not in VALID_ROLES: problems.append(f"message {i}: unknown role {role!r}") tool_calls = message.get("tool_calls") if tool_calls is not None: problems.extend( f"message {i}: {p}" for p in _validate_tool_calls(tool_calls) ) reasoning = message.get("reasoning_content") if reasoning is not None and not isinstance(reasoning, str): kind = type(reasoning).__name__ problems.append(f"message {i}: reasoning_content must be str, got {kind}") content = message.get("content") # An assistant turn that calls a tool may carry no content at all. # An empty tool_calls list calls nothing, so it gets no exception. calls_only = role == "assistant" and bool(tool_calls) if content is None: if not calls_only: problems.append(f"message {i}: content is None") elif not isinstance(content, (str, list)): kind = type(content).__name__ problems.append(f"message {i}: content must be str or list, got {kind}") elif isinstance(content, str) and not content.strip(): # A tool that returned nothing still answered its call. if not calls_only and role != "tool": problems.append(f"message {i}: content is empty or whitespace") elif isinstance(content, list): for j, part in enumerate(content): if not isinstance(part, dict) or "type" not in part: problems.append(f"message {i}: content[{j}] is not a typed part") for i, message in enumerate(conversation): if isinstance(message, dict) and message.get("role") == "system" and i != 0: problems.append(f"message {i}: system message must be first") return problemsCreate show_field_names.py. It parses a Gemma 4 reply with Gemma 4's response template and normalizes it, along with a vLLM-style message. Then it renders the three names through Qwen3's template and validates a message whose reasoning is a list. To stay short, each of the first three lines prints a message's sorted keys and its reasoning:
# show_field_names.pyfrom families import THINKING_MODELSfrom rates import QUESTIONfrom templatestudio.messages import validate_conversationfrom templatestudio.reasoning import normalize_reasoningfrom templatestudio.render import load_tokenizer, rendergemma = load_tokenizer(THINKING_MODELS["gemma4"])prompt = render(gemma, QUESTION, enable_thinking=True)raw = "<|channel>thought\nUse the rate.<channel|>About 162 yen.<turn|>"parsed = gemma.parse_response(raw, prefix=prompt)print("gemma4 parse_response:", sorted(parsed), repr(parsed["thinking"]))clean = normalize_reasoning(parsed)print("normalized: ", sorted(clean), repr(clean["reasoning_content"]))vllm_style = { "role": "assistant", "content": "About 162 yen.", "reasoning": "Use the rate.",}clean = normalize_reasoning(vllm_style)print("vllm-style normalized:", sorted(clean), repr(clean["reasoning_content"]))qwen3 = load_tokenizer(THINKING_MODELS["qwen3"])renders = set()for field in ("reasoning_content", "reasoning", "thinking"): message = {"role": "assistant", "content": "About 162 yen.", field: "x"} conversation = [*QUESTION, message] renders.add(render(qwen3, conversation, add_generation_prompt=False))print("qwen3 renders all three names the same:", len(renders) == 1)bad = {"role": "assistant", "content": "x", "reasoning_content": ["a", "list"]}print(validate_conversation([*QUESTION, bad]))Run it:
python show_field_names.pyExpected output:
gemma4 parse_response: ['content', 'role', 'thinking'] 'Use the rate.'normalized: ['content', 'reasoning_content', 'role'] 'Use the rate.'vllm-style normalized: ['content', 'reasoning_content', 'role'] 'Use the rate.'qwen3 renders all three names the same: True['message 1: reasoning_content must be str, got list']What just happened: Gemma 4's parse_response read the <|channel>thought block and returned it under thinking. normalize_reasoning moved that, and vLLM's reasoning, into reasoning_content, and kept one copy. Qwen3's template rendered all three spellings identically, because Transformers 5.22.1 copies thinking and reasoning into reasoning_content before it renders. Older Transformers releases did not alias. On 5.3.0, for example, a thinking key reached Qwen3's template unread. If you normalize once, where messages enter your code, the stored conversation is correct whichever version renders it. The last line is the new validator check, which caught a list stored where a string belongs.
Step 10: Detect missing reasoning_content in a stored conversation
Goal: add missing_reasoning, which lists the kept assistant turns in a stored conversation that carry no reasoning.
Why this step: step 6 needs the generated token IDs, and often you have only the stored conversation: a log, a replay, a bug report. From the conversation alone you can still ask whether any turn the template keeps has lost its reasoning. The check uses step 4's boundary. Only turns after the last real query count, because the template drops the rest anyway. What a lost turn looks like depends on the template. Qwen3 renders the step without a block. Qwen3.5 renders an empty block, which is what QwenLM issue #131 reported.
grid-rows: 2
grid-gap: 50
stored: "Stored conversation\nfrom a log" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
lqi: "last_query_index\nboundary" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
miss: "missing_reasoning\nkept turns, no reasoning" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
out: "[3]\nindex of the lost step" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
stored -> lqi -> miss -> out
Append missing_reasoning to templatestudio/reasoning.py:
# templatestudio/reasoning.py (add at the end)def missing_reasoning(conversation: Conversation) -> list[int]: """Indices of assistant turns that templates keep but that carry no reasoning. Only turns after the last real user query count: the template drops the reasoning of every earlier turn anyway. """ boundary = last_query_index(conversation) return [ index for index, message in enumerate(conversation) if index > boundary and message["role"] == "assistant" and not message.get("reasoning_content") ]Create show_missing_reasoning.py. The stored conversation has an earlier exchange with reasoning, then the euro question, then a tool step whose reasoning the client dropped:
# show_missing_reasoning.pyfrom families import THINKING_MODELSfrom rates import QUESTIONfrom templatestudio.reasoning import missing_reasoningfrom templatestudio.render import load_tokenizer, renderfrom templatestudio.tools import tool_call_message, tool_result_messagecall = tool_call_message("get_exchange_rate", {"base": "EUR", "quote": "JPY"})stored = [ {"role": "user", "content": "Hi."}, {"role": "assistant", "content": "Hello.", "reasoning_content": "Greet."}, *QUESTION, call, # the client dropped this step's reasoning tool_result_message("get_exchange_rate", "162.4"),]print("missing reasoning at:", missing_reasoning(stored))for family in ("qwen3", "qwen3.5"): tokenizer = load_tokenizer(THINKING_MODELS[family]) prompt = render(tokenizer, stored, enable_thinking=True) step = prompt.split("<|im_start|>assistant\n")[2] print(f"{family:8} renders the step as {step[:26]!r}")Run it:
python show_missing_reasoning.pyExpected output:
missing reasoning at: [3]qwen3 renders the step as '<tool_call>\n{"name": "get_'qwen3.5 renders the step as '<think>\n\n</think>\n\n<tool_c'What just happened: missing_reasoning ignored the greeting at index 1, which comes before the last query, and flagged the tool step at index 3. The two templates rendered that step differently. Qwen3 left the block out, and Qwen3.5 rendered it empty. Neither matches what the model wrote, which was a block full of reasoning. Read the result with the step 2 verdict in mind. With thinking off, every kept turn legitimately has no reasoning, so the check means something only for requests made with thinking on.
Step 11: Prefill the reasoning with continue_final_message
Goal: add continue_reasoning, which renders a conversation so that the model continues a reasoning block you started.
Why this step: prefilling means writing the start of the model's turn yourself, and it is how you steer a reply. Part 4's render_continuation prefills content. On a reasoning template that has a side effect: Qwen3's template closes an empty <think> block before your text, so the model skips reasoning. Transformers 5.22.1 lets continue_final_message take a field name instead of True. With "reasoning_content", the render stops inside the open block, and the model goes on thinking from your words.
direction: down
both: "Final assistant message" {
grid-columns: 2
grid-gap: 30
c: "continue_final_message=True\n<think>\\n\\n</think>\\n\\nOne euro is" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
r: "continue_final_message=\"reasoning_content\"\n<think>\\nThe user wants EUR to JPY." {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
}
Append continue_reasoning to templatestudio/reasoning.py:
# templatestudio/reasoning.py (add at the end)def continue_reasoning( tokenizer: PreTrainedTokenizerBase, conversation: Conversation) -> str: """Render with the last turn's reasoning left open for the model to go on.""" return tokenizer.apply_chat_template( conversation, tokenize=False, add_generation_prompt=False, continue_final_message="reasoning_content", )Create show_continue_reasoning.py. It prefills the reasoning, then prefills the content for comparison, and then asks to continue reasoning on a message that has none:
# show_continue_reasoning.pyimport textwrapfrom families import THINKING_MODELSfrom rates import QUESTIONfrom templatestudio.reasoning import continue_reasoningfrom templatestudio.render import load_tokenizer, render_continuationtokenizer = load_tokenizer(THINKING_MODELS["qwen3"])started = {"role": "assistant", "reasoning_content": "The user wants EUR to JPY."}print(repr(continue_reasoning(tokenizer, [*QUESTION, started])[-48:]))answer = {"role": "assistant", "content": "One euro is"}print(repr(render_continuation(tokenizer, [*QUESTION, answer])[-48:]))try: continue_reasoning(tokenizer, [*QUESTION, answer])except ValueError as error: print(textwrap.fill(f"ValueError: {error}", width=72))Run it:
python show_continue_reasoning.pyExpected output:
'rt|>assistant\n<think>\nThe user wants EUR to JPY.''_start|>assistant\n<think>\n\n</think>\n\nOne euro is'ValueError: continue_final_message is set but the final message has no"reasoning_content" to continue!What just happened: the first render ends inside an open <think> block, right after your reasoning, so the model continues it. The second render, Part 4's content prefill, closed an empty block first, and a model that reads it answers without reasoning. Prefill the content only when you want the model to skip its reasoning. The third call raised Transformers' own error, because the final message has no reasoning_content. The script prints only the last 48 characters of each render. It also wraps the error at 72 characters to fit the page; the message itself is one line.
Step 12 (optional): Run a real Qwen3 tool step through the prefix check
Goal: have Qwen3-0.6B write a real reasoned tool call on your CPU, store it with keep_reasoning, and run prefix_break on the real token IDs.
Why this step: every check so far used sample text you tokenized yourself. Re-tokenizing text does not always give back the IDs the model generated. This step uses the generated IDs directly, because those are what the server's cache sees. It needs PyTorch and the Qwen3-0.6B weights from Part 5's optional step, about 1.5 GB, and it takes about two minutes on a laptop CPU. Close other large programs first: the run needs about 3 GB of free memory.
If the weights are not cached yet, from_pretrained downloads them, about 1.5 GB, on the first run, so you do not need to go back to Part 5 for them. If you skipped Part 5's step 13, install PyTorch from its CPU wheel index inside the activated environment:
pip install "torch==2.15.0" --index-url https://download.pytorch.org/whl/cpuCreate generate_reasoned_call.py. It decodes greedily, so a run repeats exactly on one machine. generate returns token IDs rather than text, so prefix_break gets exactly what the model produced:
# generate_reasoned_call.pyimport sysimport torchfrom transformers import AutoModelForCausalLMfrom families import THINKING_MODELSfrom rates import QUESTION, get_exchange_ratefrom templatestudio.messages import Conversationfrom templatestudio.reasoning import keep_reasoning, prefix_break, split_reasoningfrom templatestudio.render import load_tokenizer, render_idsfrom templatestudio.tools import ( HermesParser, tool_call_message, tool_result_message, tool_schema,)sys.stdout.reconfigure(encoding="utf-8")tokenizer = load_tokenizer(THINKING_MODELS["qwen3"])model = AutoModelForCausalLM.from_pretrained(THINKING_MODELS["qwen3"])tools = [tool_schema(get_exchange_rate)]def generate(conversation: Conversation) -> list[int]: """Greedy-decode one assistant turn and return the new token IDs.""" prompt = render_ids(tokenizer, conversation, tools=tools) output = model.generate( torch.tensor([prompt]), max_new_tokens=700, do_sample=False, ) return output[0][len(prompt) :].tolist()request = list(QUESTION)ids = generate(request)reply = split_reasoning(tokenizer.decode(ids, skip_special_tokens=True))print(f"generated {len(ids)} tokens, reasoning {len(reply.reasoning)} chars")call = HermesParser().parse(reply.answer)[0]print("parsed:", call.name, call.arguments)result = tool_result_message(call.name, str(get_exchange_rate(**call.arguments)))step = tool_call_message(call.name, call.arguments)for label, message in (("kept", keep_reasoning(step, reply)), ("dropped", step)): next_request = [*request, message, result] where = prefix_break(tokenizer, request, ids, next_request, tools=tools) print(f"{label:8} break={where}")final = generate([*request, keep_reasoning(step, reply), result])text = tokenizer.decode(final, skip_special_tokens=True)print("answer:", split_reasoning(text).answer)Run it:
python generate_reasoned_call.pyExpected output:
generated 156 tokens, reasoning 540 charsparsed: get_exchange_rate {'base': 'EUR', 'quote': 'JPY'}kept break=Nonedropped break=231answer: One euro is approximately **162.4 yen**.The run prints a Loading weights progress bar on standard error before this output. The lines to check are the two break= lines. The token count, the reasoning length and the wording of the answer can differ on other hardware.
What just happened: Qwen3-0.6B generated 156 tokens, counting the call and its end-of-turn token, and wrote Part 5's tool call. Stored with keep_reasoning, the step re-rendered to exactly the IDs the model generated, so request 2 extended request 1 with no break. Stored the Part 5 way, without the reasoning, request 2 broke at token 231, the first token the model wrote. None of the 156 generated tokens could be reused. The final answer came from the request with the reasoning kept. This is the full loop from Part 5, and now it is cache-friendly. In your own agent, call prefix_break after each step in a test run, and fail the test when it returns the first generated token.
Fix "has no reasoning_content to continue" and other reasoning errors
| Error or symptom | Cause | Fix |
|---|---|---|
ValueError: continue_final_message is set but the final message has no "reasoning_content" to continue! | continue_reasoning on a message with no reasoning_content key (step 11); an empty string is accepted and renders an open <think> block | Put the prefill text in reasoning_content, or prefill the content with render_continuation |
ValueError: continue_final_message is set to "plan" but this is not an accepted field in the chat_template | continue_final_message was given a field the template never reads | Use "reasoning_content" or True |
ValueError: reply stopped inside its reasoning block; do not store it | keep_reasoning got a reply cut off by max_new_tokens | Raise max_new_tokens, or retry; do not store the half block |
InvalidConversation: message 1: reasoning_content must be str, got list from safe_render | Reasoning stored as a list, often a provider's list of content parts (step 9) | Join the parts into one string before storing |
enable_thinking=False has no effect | The template ignores the flag, as both Qwen3 2507 templates and Llama 3 do (step 2) | Pick the Instruct or Thinking checkpoint; the checkpoint is the switch |
A misspelled flag such as enable_thinkng changes nothing, with no warning | Unused template variables are ignored (step 1) | Run thinking_default once per model in CI |
| Qwen3.5 returns no reasoning | Its template defaults to thinking off (step 2) | Pass enable_thinking=True |
| The answer shown to the user is empty | The reply hit max_new_tokens inside its block; split_reasoning returned closed=False | Check reply.closed before displaying, and raise the limit |
prefix_break returns the first generated token | The stored step lost its reasoning (step 6) | Store it with keep_reasoning |
prefix_break returns a token just after <think> | The model's spacing differs from the template's re-render (step 6) | Your stored message cannot change it; measure it once per model and expect it |
AttributeError: This tokenizer does not have a `response_template` for parsing chat responses! | parse_response called on Qwen3 instead of Gemma 4 in step 9; Qwen3 ships no response template | Split Qwen3 output with split_reasoning, or parse with Part 5's parse_tool_calls and a fallback parser |
| Step 12 prints nothing after the progress bar and the shell reports the process was killed | Less than about 3 GB of free memory for the weights and the generation | Close other large programs and run it again |
IndexError: list index out of range at HermesParser().parse(reply.answer)[0] | The model answered without calling a tool, so the parser returned an empty list | Check the list before indexing, or use Part 5's parse_tool_calls, which raises UnparsedToolCall for text that looks like a call |
| The reasoning field reads as an empty string from vLLM | vLLM 0.16 and later return reasoning, not reasoning_content | Run every incoming message through normalize_reasoning (step 9) |
Empty <think> blocks in rendered history on Qwen3.5 | Kept turns stored without reasoning (step 10, QwenLM #131) | Store reasoning with keep_reasoning; check stored logs with missing_reasoning |
prefix_break returns a number in the middle of a tool loop
Compare the number with the length of request 1's prompt. If it is the first generated token, the stored step lost its reasoning: build it with keep_reasoning. If it is one or two tokens later, the model's spacing differs from the template's, as in step 6's no-newline case. If it is a few tokens before the end of the prompt, with thinking off on Qwen3, it is the expected break from step 8.
How the reasoning module fits into the Transformers render loop
reasoning.py is the only new module. It sits beside Part 5's tools.py and uses render.py for every render. The diagram uses the names from your code.
direction: down
loop: "generate_reasoned_call.py: your agent loop" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
out: "reasoning.py, way out\nsplit_reasoning -> Reply" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
tools: "tools.py (Part 5)\nHermesParser, tool_call_message" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
store: "reasoning.py, storage\nkeep_reasoning, normalize_reasoning" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
checks: "reasoning.py, checks\nthinking_default, prefix_break,\nlast_query_index, missing_reasoning" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
rend: "render.py\nrender / render_ids / safe_render\n(**template_kwargs)" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
act: "apply_chat_template\nthe model's chat template" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
loop -> out: "1. generated text"
out -> tools: "2. answer"
tools -> store: "3. message"
out -> store: "3. reasoning"
store -> rend: "4. next request"
checks -> rend: "probe renders"
rend -> act: "5. enable_thinking, tools"
Follow the numbers. The yellow box splits what the model wrote. The answer goes to Part 5's parser, and the reasoning joins the message Part 5 builds, in the blue box. The next request goes through render.py to the template. The purple checks call the same render and render_ids, so they test exactly what your loop sends. continue_reasoning calls apply_chat_template directly, because no other function prefills a field. reasoning.py depends on render.py and messages.py, and nothing imports it except your scripts.
Complete code: handle reasoning in a chat template in Python
The project after this part, with the files this part added or changed marked. In the listings below, a step number inside a code comment refers to the part that wrote that file, not to this page, because the listings match the files you already have:
chat-templates-part1/├── templatestudio/│ ├── __init__.py│ ├── messages.py (Message gains reasoning_content)│ ├── render.py (render, render_ids, safe_render forward kwargs)│ ├── jinja_env.py (only if you built Part 3; unused here)│ ├── anatomy.py (only if you built Part 4; unused here)│ ├── tools.py (unchanged since Part 5)│ └── reasoning.py (new)├── families.py (THINKING_MODELS appended)├── fetch_models.py (only if you built Part 4)├── fetch_thinking.py (new)├── rates.py (Part 5)├── samples.py (QWEN3_REASONED_CALL appended)├── show_thinking_prompt.py├── compare_thinking.py├── show_split.py├── show_history_rule.py├── show_keep_reasoning.py├── show_prefix_break.py├── show_replay_strip.py├── show_expected_breaks.py├── show_field_names.py├── show_missing_reasoning.py├── show_continue_reasoning.py├── generate_reasoned_call.py (optional, needs torch)└── ... Part 5's scriptstemplatestudio/reasoning.py:
# templatestudio/reasoning.pyfrom __future__ import annotationsfrom dataclasses import dataclassfrom typing import castfrom transformers import PreTrainedTokenizerBasefrom templatestudio.messages import Conversation, Messagefrom templatestudio.render import first_divergence, render, render_idsOPEN, CLOSE = "<think>", "</think>"# The names reasoning travels under: the same names, in the same order of# preference, as Part 3's jinja_env.py.REASONING_FIELDS = ("reasoning_content", "thinking", "reasoning")def thinking_default( tokenizer: PreTrainedTokenizerBase, conversation: Conversation, flag: str = "enable_thinking",) -> str: """Report what a template does with a thinking flag. It renders the generation prompt three times: without the flag, with flag=True and with flag=False, and compares the three strings. """ unset = render(tokenizer, conversation) on = render(tokenizer, conversation, **{flag: True}) off = render(tokenizer, conversation, **{flag: False}) if on == off: return "ignored" if unset == on: return "default on" if unset == off: return "default off" return "unset differs from both"@dataclass(frozen=True)class Reply: """One assistant turn, split into its reasoning and its visible answer.""" reasoning: str answer: str closed: bool # False when the output stopped inside the reasoning blockdef split_reasoning(text: str, *, prompt_opened: bool = False) -> Reply: """Split raw model output at its reasoning block. Set prompt_opened when the generation prompt already wrote <think>, so the output starts inside the block and only the closing tag appears. """ if prompt_opened and OPEN not in text: text = OPEN + text start = text.find(OPEN) if start == -1: return Reply("", text.strip(), closed=True) end = text.find(CLOSE, start) if end == -1: return Reply(text[start + len(OPEN) :].strip("\n"), "", closed=False) reasoning = text[start + len(OPEN) : end].strip("\n") return Reply(reasoning, text[end + len(CLOSE) :].strip(), closed=True)def last_query_index(conversation: Conversation) -> int: """Index of the last real user turn, the way Qwen3's template finds it. A user message that only wraps a tool result in <tool_response> tags is not a query, so the walk skips it. """ for index in range(len(conversation) - 1, -1, -1): message = conversation[index] content = message.get("content") if message["role"] != "user" or not isinstance(content, str): continue wrapped = content.startswith("<tool_response>") and content.endswith( "</tool_response>" ) if not wrapped: return index return len(conversation) - 1def keep_reasoning(message: Message, reply: Reply) -> Message: """Return message with the reply's reasoning where templates read it.""" if not reply.closed: raise ValueError("reply stopped inside its reasoning block; do not store it") return {**message, "reasoning_content": reply.reasoning}def prefix_break( tokenizer: PreTrainedTokenizerBase, request: Conversation, generated_ids: list[int], next_request: Conversation, *, tools: list[dict] | None = None, **template_kwargs: object,) -> int | None: """First token where next_request stops repeating what the model saw. The model saw request's prompt and then wrote generated_ids. If the next prompt starts with exactly those tokens, a prefix cache can reuse them and this returns None. Otherwise it returns the first index that differs. """ seen = render_ids(tokenizer, request, tools=tools, **template_kwargs) seen = seen + generated_ids sent = render_ids(tokenizer, next_request, tools=tools, **template_kwargs) return first_divergence(seen, sent[: len(seen)])def normalize_reasoning(message: dict) -> Message: """Store reasoning found under any known name as reasoning_content. The first non-empty field in REASONING_FIELDS wins; the other names are dropped, so the message carries one copy under one name. """ found = [message[f] for f in REASONING_FIELDS if message.get(f)] clean = {k: v for k, v in message.items() if k not in REASONING_FIELDS} if found: clean["reasoning_content"] = found[0] return cast(Message, clean)def missing_reasoning(conversation: Conversation) -> list[int]: """Indices of assistant turns that templates keep but that carry no reasoning. Only turns after the last real user query count: the template drops the reasoning of every earlier turn anyway. """ boundary = last_query_index(conversation) return [ index for index, message in enumerate(conversation) if index > boundary and message["role"] == "assistant" and not message.get("reasoning_content") ]def continue_reasoning( tokenizer: PreTrainedTokenizerBase, conversation: Conversation) -> str: """Render with the last turn's reasoning left open for the model to go on.""" return tokenizer.apply_chat_template( conversation, tokenize=False, add_generation_prompt=False, continue_final_message="reasoning_content", )templatestudio/render.py:
# templatestudio/render.pyfrom __future__ import annotationsfrom transformers import AutoTokenizer, PreTrainedTokenizerBasefrom templatestudio.messages import Conversation, assert_validdef load_tokenizer(model_id: str) -> PreTrainedTokenizerBase: """Load a tokenizer together with the chat template it ships with.""" return AutoTokenizer.from_pretrained(model_id)def render( tokenizer: PreTrainedTokenizerBase, conversation: Conversation, *, add_generation_prompt: bool = True, tools: list[dict] | None = None, **template_kwargs: object,) -> str: """Render a conversation into the model's wire format, using its own template. tools is a list of JSON tool schemas. A template that has no tools branch ignores it without an error. template_kwargs, such as enable_thinking=False, become variables the template can read; one it never reads is ignored. """ return tokenizer.apply_chat_template( conversation, tools=tools, tokenize=False, add_generation_prompt=add_generation_prompt, **template_kwargs, )def visible_whitespace(text: str) -> str: """Show spaces, tabs and newlines so two renders can be compared by eye.""" return ( text.replace("\t", "[TAB]") .replace(" ", "·") .replace("\n", "\\n\n") )def first_divergence(a: str | list[int], b: str | list[int]) -> int | None: """Return the first index where a and b differ, or None if they are equal. Works on strings (characters) and on lists of token IDs alike. """ for index, (item_a, item_b) in enumerate(zip(a, b)): if item_a != item_b: return index if len(a) != len(b): return min(len(a), len(b)) return Nonedef render_ids( tokenizer: PreTrainedTokenizerBase, conversation: Conversation, *, add_generation_prompt: bool = True, tools: list[dict] | None = None, **template_kwargs: object,) -> list[int]: """Render a conversation straight to the token IDs the model receives.""" encoded = tokenizer.apply_chat_template( conversation, tools=tools, tokenize=True, add_generation_prompt=add_generation_prompt, **template_kwargs, ) return encoded["input_ids"]def safe_render( tokenizer: PreTrainedTokenizerBase, conversation: Conversation, *, add_generation_prompt: bool = True, tools: list[dict] | None = None, **template_kwargs: object,) -> str: """Check the input contract, then render. Use this in production.""" assert_valid(conversation) return render( tokenizer, conversation, add_generation_prompt=add_generation_prompt, tools=tools, **template_kwargs, )def render_continuation( tokenizer: PreTrainedTokenizerBase, conversation: Conversation,) -> str: """Render with the final assistant turn left open, so the model continues it.""" if not conversation or conversation[-1]["role"] != "assistant": raise ValueError("render_continuation needs an assistant message last") return tokenizer.apply_chat_template( conversation, tokenize=False, add_generation_prompt=False, continue_final_message=True, )templatestudio/messages.py:
from __future__ import annotationsfrom typing import Literal, NotRequired, TypedDict, UnionRole = Literal["system", "developer", "user", "assistant", "tool"]VALID_ROLES: frozenset[str] = frozenset( {"system", "developer", "user", "assistant", "tool"})# A content part is one item in a multimodal or tool message.ContentPart = dict[str, object]Content = Union[str, list[ContentPart]]# One entry in an assistant message's tool_calls list (step 6).ToolCall = dict[str, object]class Message(TypedDict): role: Role content: NotRequired[Content | None] tool_calls: NotRequired[list[ToolCall]] reasoning_content: NotRequired[str]Conversation = list[Message]def validate_conversation(conversation: Conversation) -> list[str]: """Return every input-contract violation. An empty list means well-formed.""" if not conversation: return ["conversation is empty"] problems: list[str] = [] for i, message in enumerate(conversation): if not isinstance(message, dict): kind = type(message).__name__ problems.append(f"message {i}: must be a dict, got {kind}") continue role = message.get("role") if role not in VALID_ROLES: problems.append(f"message {i}: unknown role {role!r}") tool_calls = message.get("tool_calls") if tool_calls is not None: problems.extend( f"message {i}: {p}" for p in _validate_tool_calls(tool_calls) ) reasoning = message.get("reasoning_content") if reasoning is not None and not isinstance(reasoning, str): kind = type(reasoning).__name__ problems.append(f"message {i}: reasoning_content must be str, got {kind}") content = message.get("content") # An assistant turn that calls a tool may carry no content at all. # An empty tool_calls list calls nothing, so it gets no exception. calls_only = role == "assistant" and bool(tool_calls) if content is None: if not calls_only: problems.append(f"message {i}: content is None") elif not isinstance(content, (str, list)): kind = type(content).__name__ problems.append(f"message {i}: content must be str or list, got {kind}") elif isinstance(content, str) and not content.strip(): # A tool that returned nothing still answered its call. if not calls_only and role != "tool": problems.append(f"message {i}: content is empty or whitespace") elif isinstance(content, list): for j, part in enumerate(content): if not isinstance(part, dict) or "type" not in part: problems.append(f"message {i}: content[{j}] is not a typed part") for i, message in enumerate(conversation): if isinstance(message, dict) and message.get("role") == "system" and i != 0: problems.append(f"message {i}: system message must be first") return problemsdef _validate_tool_calls(tool_calls: object) -> list[str]: """Check the shape of an assistant message's tool_calls.""" if not isinstance(tool_calls, list): return ["tool_calls must be a list"] problems: list[str] = [] for j, call in enumerate(tool_calls): function = call.get("function") if isinstance(call, dict) else None if not isinstance(function, dict): problems.append(f"tool_calls[{j}]: missing function object") elif "name" not in function or "arguments" not in function: problems.append( f"tool_calls[{j}]: function needs name and arguments" ) return problemsclass InvalidConversation(ValueError): """Raised when a conversation violates the input contract."""def assert_valid(conversation: Conversation) -> None: """Raise InvalidConversation listing every violation, or return cleanly.""" problems = validate_conversation(conversation) if problems: raise InvalidConversation("; ".join(problems))families.py:
# families.py# Three families, three wire formats. The Llama 3 and Gemma 3 repos are ungated# copies whose chat templates match the official, gated repos exactly.FAMILIES: dict[str, str] = { "qwen3": "Qwen/Qwen3-0.6B", "llama3": "NousResearch/Meta-Llama-3-8B-Instruct", "gemma3": "unsloth/gemma-3-1b-it",}# Gemma 4 is ungated. It joins in step 6, where its turn markers matter.GEMMA4 = "google/gemma-4-E2B-it"# Part 6: templates that handle reasoning, plus one family that has none.THINKING_MODELS: dict[str, str] = { "qwen3": "Qwen/Qwen3-0.6B", "qwen3.5": "Qwen/Qwen3.5-0.8B", "qwen3-instruct-2507": "Qwen/Qwen3-4B-Instruct-2507", "qwen3-thinking-2507": "Qwen/Qwen3-4B-Thinking-2507", "gemma4": GEMMA4, "llama3": FAMILIES["llama3"],}samples.py:
# samples.py# Raw text in each family's tool-call format, as a model writes it after its# generation prompt. Step 13 has Qwen3 write a real call instead.QWEN3_CALL = ( "<tool_call>\n" '{"name": "get_exchange_rate", "arguments": {"base": "EUR", "quote": "JPY"}}\n' "</tool_call>")FENCED_CALL = ( "```json\n" '{"name": "get_exchange_rate", "arguments": {"base": "EUR", "quote": "JPY"}}\n' "```")GEMMA4_CALL = ( '<|tool_call>call:get_exchange_rate{base:<|"|>EUR<|"|>,quote:<|"|>JPY<|"|>}' "<tool_call|>")PLAIN_ANSWER = "One euro buys about 162.4 yen."# Part 6: one reasoned tool step, as Qwen3 writes it with thinking on.QWEN3_REASONING = "The user wants EUR to JPY. I should call get_exchange_rate."QWEN3_REASONED_CALL = ( "<think>\n" + QWEN3_REASONING + "\n</think>\n\n" + QWEN3_CALL + "<|im_end|>")tools.py and rates.py are unchanged; their listings are in Part 5's complete code. The scripts are listed in full in the steps that create them.
Next steps: extend the reasoning module to preserved thinking, Harmony and a server
- Preserved thinking on Qwen3.6. Qwen3.6's template reads a second flag,
preserve_thinking, which keeps reasoning across user turns and should remove step 8's first break. Add the model toTHINKING_MODELS, runthinking_default(tokenizer, conversation, flag="preserve_thinking"), and rerunshow_expected_breaks.py. That measures what keeping all reasoning costs in prompt length against what it saves in cache reuse. - gpt-oss and Harmony channels. gpt-oss writes reasoning in a separate
analysischannel of the Harmony format, not in<think>tags. Write asplit_reasoningvariant for its channel markers and pointprefix_breakat it. A vLLM pull request, #62786, found its/tokenizeoutput and the Harmony library differing at token 82, so expect the same kind of spacing break as in step 6. - A server-side check. Send the same tool loop to a local vLLM or SGLang server with prefix caching on, and compare its cached-token count for request 2 with
prefix_break. The two should agree; Part 10 makes the same comparison token for token against llama-server. If they do not, the server is changing your messages, as in vLLM issue #38488.
The Chat Templates Handbook, the book this series accompanies, covers reasoning templates in chapter 6. That includes the token-cost arithmetic of dropping reasoning, and the history rule this part calls interleaved thinking, which the book names the reasoning checkpoint. Part 7 turns to multimodal templates: how a chat template renders images and other non-text content parts of a message.
References
- Hugging Face Transformers v5.22.1: chat templates, including
continue_final_messagewith a field name. - Hugging Face Transformers v5.22.1: response parsing,
parse_responseand theprefix=argument. chat_template_utils.pyat transformers v5.22.1, the reasoning-field aliasing and thecontinue_final_messageerrors.- Transformers v5.22.2 release and PR #50431, the Jinja sandbox hardening.
- Qwen/Qwen3-0.6B model card:
enable_thinkingand "No Thinking Content in History"; itstokenizer_config.jsonholds the chat template withns.last_query_index. - The pinned packages: tokenizers 0.23.3, huggingface-hub 2.4.0, Jinja2 3.1.6 and transformers 5.22.1.
chat_template_utils.pyat transformers v5.3.0, a release without reasoning-field aliasing.- Qwen/Qwen3.5-0.8B, Qwen/Qwen3-4B-Instruct-2507 and Qwen/Qwen3-4B-Thinking-2507 model cards.
- google/gemma-4-E2B-it, its chat template and response template.
- Qwen/Qwen3.6-35B-A3B,
preserve_thinkingand the latest-query rule on its model card. - Fireworks: thinking history, interleaved and preserved thinking.
- vLLM: reasoning outputs, the
reasoningfield; RFC #27755 and PR #33402, the rename and the removal ofreasoning_content. - vLLM issue #38488 and PR #42664, incoming
reasoning_contentdropped and then normalized. - vLLM: interleaved thinking and Anthropic: thinking in tool workflows, the generation-time sense of the term.
- vLLM: automatic prefix caching, the server-side reuse
prefix_breakpredicts. - SGLang: separate reasoning, the
reasoning_contentfield and its parsers. - opencode issue #19081, cache invalidation from stripped reasoning.
- QwenLM/Qwen3.8 issue #131, empty historical
<think>blocks. - Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, 2023, the vLLM paper.
- Zheng et al., SGLang: Efficient Execution of Structured Language Model Programs, 2023, RadixAttention prefix reuse.
- Qwen3 Technical Report, 2025, thinking and non-thinking modes.
- OpenAI Harmony format, gpt-oss reasoning channels; vLLM PR #62786, the token-82 difference.
- PyTorch 2.15.0 CPU wheels, the index for the optional step 12 install.
Related Articles
- How to Validate Chat Messages Before apply_chat_template
- How to Render a Chat Template with Jinja Like Transformers
- How to See What apply_chat_template Sends Your Model
- How to Pass Tools to apply_chat_template and Parse the Calls
