How to Gate Chat Template Changes in CI with One Audit

What you'll build: one CI gate for every chat template you serve

This tutorial builds a chat template CI gate: one Python audit that checks every Hugging Face chat template you serve and fails the pull request when one regresses.

A pull request adds two models to a serving fleet that already runs Qwen3. One command audits all three. One of the new models fails:

text

== qwen3template audit: PASS  [reviewed] default: long literal string injected into output  [reviewed] default: branch on message content - possible trigger condition== smollm3template audit: FAIL  [portability] default: line 53: string filter: fails on a dict, prints a list as      JSON  [portability] default: line 61: string filter: fails on a dict, prints a list as      JSON  [security] default: long literal string injected into output  [security] default: branch on message content - possible trigger condition  [parity] tool_round_trip: huggingface-jinja: Error: Unknown ObjectValue filter:      string== gemma3template audit: PASSexit code: 1

This intermediate tutorial is for engineers who serve open-weight models with Hugging Face Transformers and want apply_chat_template regressions, whitespace changes and tampered templates caught in a pull request, not in production. If you have not seen why a chat template is the last unowned layer in the stack, that article is the background. Budget about 45 minutes. Everything runs on a CPU with no model weights, no API key and no Hugging Face token. It also runs offline, except for two tokenizer downloads in step 11 if they are not already in your cache.

That output is the gate you build here. It loads the tokenizer folder each service ships and runs the template checks from three earlier parts: the fingerprint pin and backdoor scan from Part 12, the golden-token tests from Part 11, and the portability lint and Jinja engine parity from Part 9. It prints each finding with its location and exits 1 if any finding fails the build.

SmolLM3 fails on a string filter that the JavaScript Jinja engine cannot apply to a dictionary, and on scan findings nobody has reviewed yet. Gemma 3 passes, and adding it took one recording command, one pin and one config entry.

By the end you will have templatestudio/studio.py: an audit_template that collects every template-level finding into one report, a ci_gate that turns the report into an exit code, and a ModelGate record, one per model, which is what your application passes across the seam. The CI script adds only two settings: which Jinja engines to compare, and the warning policy. Along the way the gate fails on three edits a pull request could make, and on Part 1's f-string wrapper. Then you add a second and a third model, and set a severity policy that can downgrade portability findings to warnings but can never downgrade a security finding.

Verified against Python 3.12.2, transformers==5.22.1, tokenizers==0.23.3, huggingface-hub==2.4.0, jinja2==3.1.6, Node.js 20.20.0 and @huggingface/jinja==0.5.12 on 2026-11-12. Every step ran on a CPU with no model weights loaded. The GitHub Actions workflow was checked as YAML and its gate command was run locally under both shells; it was not run on GitHub.

This is Part 13, the last part of Chat Templates in Practice. It continues the project from Part 12, How to Detect Chat Template Backdoors and Forged Tokens. Twelve parts each built one check. This part runs them as one.

Prerequisites for the chat template CI gate: the project from Parts 1 to 12

You need the chat-templates-part1/ project with its virtual environment, as Part 12 left it. This part imports these files and changes only security_pins.py:

templatestudio/render.py (Part 1, last changed in Part 6) and naive.py (Part 1): load_tokenizer, render, build_prompt_naive
templatestudio/messages.py (Part 2, last changed in Part 7) and templatestudio/jinja_env.py (Part 3)
templatestudio/tools.py (Part 5): tool_schema, tool_call_message, tool_result_message
templatestudio/parity.py, engines/hf_jinja.mjs, package.json, package-lock.json and node_modules/, amounts.py (Part 9) and rates.py (Part 5): lint_portability, chat_variables, compare_engines, EngineError, the EngineRenderer interface, the CPythonJinja and NodeJinja engines, and Part 9's currency call (ARGUMENTS, convert, convert_loop). rates.py is listed because amounts.py imports it
templatestudio/serving.py and session.py (Part 10): TokenRenderer, the interface for anything that turns a conversation into token IDs, and TransformersTokens, which does it with apply_chat_template
templatestudio/golden.py, golden_cases.py, goldens/qwen3.json, the qwen3-tokenizer/ folder, golden_ship.py and whitespace_commit.py (Part 11): GoldenCase, GoldenFailure, CHAR_WINDOW, check_goldens, load_goldens, record_goldens, record_golden_ids, save_goldens; SHIPPED (the path qwen3-tokenizer, the folder your service loads) and CASES, the six golden conversations; HUNKS, Part 11's three-hunk whitespace tidy-up, and apply_hunks. golden_ship.py writes the folder again from the cached Qwen/Qwen3-0.6B tokenizer, so it works offline
templatestudio/security.py, security_pins.py and backdoors.py (Part 12): template_fingerprint, template_sources, assert_pinned, scan_template, QWEN3_PINS (the full 64-character SHA-256 of each template; messages print its first 12 characters), QWEN3_REVIEWED; and the refund backdoor, REFUND_TRIGGER and insert_backdoor

If a file is missing, copy it from the complete code of the part that last changed it. This part adds one module, templatestudio/studio.py, two files your application owns (studio_models.py and the CI entry point studio_gate.py), ten more studio_ scripts that the steps run, and a GitHub Actions workflow.

Go to the project folder, activate the environment, and set the two variables every part has used. Run the cd from the folder that contains chat-templates-part1/:

bash

cd chat-templates-part1source .venv/bin/activateexport TRANSFORMERS_NO_ADVISORY_WARNINGS=1export HF_HUB_OFFLINE=1

On Windows, activate with .venv\Scripts\activate in PowerShell or source .venv/Scripts/activate in Git Bash. In PowerShell, set the variables with $env:TRANSFORMERS_NO_ADVISORY_WARNINGS = "1" and $env:HF_HUB_OFFLINE = "1". Run every command from the project root. Several steps check an exit code with echo $?; in PowerShell the same number is in $LASTEXITCODE. Where a command ends in > /dev/null, use > $null in PowerShell, and where a command is split over two lines with \, type it as one line.

The pins are unchanged from Part 12: Python 3.12 (verified on 3.12.2), transformers 5.22.1, tokenizers 0.23.3, huggingface-hub 2.4.0 and jinja2 3.1.6, plus Node.js and @huggingface/jinja 0.5.12 from Part 9. This part installs nothing. Transformers 5.22.2 shipped on 2026-11-06 with a Jinja sandbox fix, "Harden our Jinja sandboxing" (PR #50431), for a CPython bug that upstream fixed only on Python 3.13 and later. Stay on 5.22.1 to match the numbers here, and treat the bump as the first change you run through the finished gate. Check the versions before step 1:

bash

python -c "import transformers as t, jinja2 as j; print(t.__version__, j.__version__)"node --version

text

5.22.1 3.1.6v20.20.0

Your Node version can differ. @huggingface/jinja 0.5.12 needs Node 18 or later. This was verified on Node 20.20.0, and Node 20 reached end of life on 2026-04-30, so use the current LTS, Node 26.x, for a new install and in CI (Node.js release schedule). The workflow's Node 26 was not run here.

Steps 1 to 10 need no download; they use only files already on your disk. Step 11 saves two more tokenizers, HuggingFaceTB/SmolLM3-3B (used in Part 9) and unsloth/gemma-3-270m-it (used in Part 10). If either is no longer in your Hugging Face cache, run unset HF_HUB_OFFLINE (PowerShell: Remove-Item Env:HF_HUB_OFFLINE) before step 11 so they can download. Both are tokenizer files only, about 50 MB together on disk after saving, with no model weights.

Code blocks that edit an existing file start with a comment such as # templatestudio/studio.py (add at the end) or (replace ci_gate). That line tells you where the code goes. It is not part of the code, so do not paste it. A block that starts with only a file name, such as # studio_gate.py, is the whole file: create it, or replace the file if it already exists.

How to structure a chat template CI gate: one ModelGate record per model

Every part of this series built a check for one way a chat template goes wrong. Each check ran from its own script, and only when someone remembered to run it. A check like that protects nothing. This part puts the checks behind one seam that chapter 13 of The Chat Templates Handbook calls the integration contract.

The toolkit owns one job: take the tokenizer your service loads and prove that its templates still produce the reviewed, tested, portable wire format. Your application owns everything else: which models it serves, the conversations it sends, the server, and what it does with the output. The seam between them is one record per model. You name the tokenizer folder, the goldens recorded for it, the pins, and the findings someone reviewed. The toolkit hands back one report and one exit code.

direction: down
classes: {
  app: {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
  seam: {style: {fill: "#FFD93D"; stroke: "#C9A92A"; font-color: "#2C2C2A"}}
  check: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  gate: {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
  beside: {style: {fill: "#95A5A6"; stroke: "#6F7C7D"; font-color: "#FFFFFF"}}
}
app: "Your application:\nmodels it serves, conversations,\nserver, what it does with output" {class: app}
seam: "The seam: one ModelGate per model\ntokenizer folder, goldens, pins, reviews" {class: seam}
studio: "Template Studio: audit_template" {
  grid-rows: 2
  grid-columns: 3
  label.near: top-center
  prov: "provenance\nPart 12" {class: check}
  sec: "security\nPart 12" {class: check}
  port: "portability\nPart 9" {class: check}
  gold: "golden\nPart 11" {class: check}
  par: "parity\nPart 9" {class: check}
}
gate: "ci_gate: exit 0 or exit 1" {class: gate}
beside: "Beside the gate, not inside it:\nsystem handling (Part 4),\nstop set (Part 8),\nprefix stability (Part 10)" {class: beside}
app -> seam: "names the model"
seam -> studio: "audit_model"
studio -> gate: "one report"

Read the diagram from the top. Your application crosses the seam with one ModelGate per model. The CI script adds two settings, the parity engines and the warning policy. The five checks below it are the series' template checks. Each one writes findings of its own kind (a scan finding can also come back as reviewed), and ci_gate turns the whole report into the one number CI understands.

The grey box is the boundary of this gate. Three earlier checks belong beside it as separate CI steps, because each needs something the gate does not have. Part 8's check_stop_alignment reads the model's generation config. Part 10's assert_prefix_stable needs a recorded multi-request session. Part 4's system_handling does read the template, but its right answer (own turn, folded or dropped) is a policy you choose per model family, so it needs that expected answer as its fixture. This tutorial does not wire those three. The Next section at the end names the fixture each one needs.

The contract has one practical test. Adding a model, a tool or a new kind of message must be a new entry or a new golden case, never a new rendering path. Step 11 measures that for two models.

Step 1: Define a report that every chat template check writes to

Goal: add templatestudio/studio.py with a Finding and a StudioReport that decides pass or fail.

Why this step: the checks you built return five different shapes. lint_portability returns strings like line 53: .... scan_template returns finding names, check_goldens returns GoldenFailure objects, compare_engines returns ParityReport objects, and assert_pinned raises a ValueError. A gate that knows five shapes has to change every time a check changes. So every check writes the same small record: which check, where, and what. The report then needs one rule. It passes when it holds nothing except findings someone has already reviewed.

The import block is the module's complete list. Steps 2 to 7 append functions that use these names, and step 13 replaces ci_gate and adds an ALWAYS_BLOCK constant above it, so write the imports once now.

python

# templatestudio/studio.py"""One audit over a tokenizer's chat templates, and one exit code for CI."""from __future__ import annotationsimport textwrapfrom collections.abc import Collection, Mappingfrom dataclasses import dataclass, fieldfrom transformers import PreTrainedTokenizerBasefrom templatestudio.golden import (    CHAR_WINDOW,    GoldenCase,    GoldenFailure,    check_goldens,    load_goldens,)from templatestudio.parity import (    EngineError,    EngineRenderer,    chat_variables,    compare_engines,    lint_portability,)from templatestudio.security import (    assert_pinned,    scan_template,    template_fingerprint,    template_sources,)from templatestudio.render import load_tokenizerfrom templatestudio.serving import TokenRenderer, TransformersTokens@dataclass(frozen=True)class Finding:    check: str  # provenance, security, portability, golden, parity or reviewed    where: str  # the template or golden case the finding is about    detail: str@dataclassclass StudioReport:    findings: list[Finding] = field(default_factory=list)    @property    def failed(self) -> list[Finding]:        """Everything except findings someone reviewed for this exact template."""        return [f for f in self.findings if f.check != "reviewed"]    @property    def ok(self) -> bool:        return not self.failed

Add a script that builds a report by hand:

python

# studio_report.py"""What makes a report pass: no finding outside the reviewed ones."""from templatestudio.studio import Finding, StudioReportreport = StudioReport()print("empty report ok:", report.ok)report.findings.append(Finding("reviewed", "default", "long literal string"))print("reviewed finding only, ok:", report.ok)report.findings.append(Finding("golden", "plain", "token 9"))print("plus a golden finding, ok:", report.ok, "| failed:", len(report.failed))

Run it:

bash

python studio_report.py

Expected output:

text

empty report ok: Truereviewed finding only, ok: Trueplus a golden finding, ok: False | failed: 1

What just happened: a report with only a reviewed finding still passes. This is how the gate will live with the two findings Part 12 recorded for Qwen3's stock template. One golden finding fails it, and failed holds exactly that finding. Every later step produces Finding records and adds them to this report.

Step 2: Lint and scan a chat template with reviews bound to its fingerprint

Goal: add template_findings, which runs Part 9's portability lint and Part 12's scan on one template and accepts reviewed findings only for the exact template that was reviewed.

Why this step: Part 12's scan flags stock Qwen3 twice: a long literal (the tool-calling instructions) and a branch on message content (its <tool_response> and </think> tests). Both are legitimate, and someone read them. A gate that fails on them forever is a gate the team learns to ignore. A gate that ignores those two finding names forever is worse. In Part 12's step 6, the refund backdoor's content branch reported the same finding name as Qwen3's own two branches, so it hid behind a finding someone had already accepted. So the review is keyed by the template's SHA-256 fingerprint. A finding counts as reviewed only when it is on the list for that exact source. If anyone changes one byte of the template, the review no longer applies and the findings fail again until someone reads the new template. Part 12 called the scan a tripwire, not a gate, and on its own it still is one. The fingerprint is what makes it safe to fail a build on.

direction: down
classes: {
  data: {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
  act: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  ok: {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
  bad: {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
}
src: "Template source" {class: data}
fp: "SHA-256 fingerprint" {class: act}
rev: "reviewed\n{fingerprint: [findings]}" {class: data}
scan: "scan_template finding" {class: act}
q: "On the list for\nthis fingerprint?" {class: act}
yes: "reviewed\n(does not fail)" {class: ok}
no: "security\n(fails the build)" {class: bad}
src -> fp
src -> scan
fp -> rev: "look up"
rev -> q
scan -> q
q -> yes: "yes"
q -> no: "no"

The fingerprint is the only key into the review record. A finding's name alone never reaches the green box.

python

# templatestudio/studio.py (add at the end)def template_findings(    name: str, source: str, reviewed: Mapping[str, Collection[str]]) -> list[Finding]:    """Lint and scan one template. reviewed maps a template fingerprint to    the scan findings someone read and accepted for exactly that source."""    accepted = reviewed.get(template_fingerprint(source), ())    findings = [Finding("portability", name, f) for f in lint_portability(source)]    for finding in scan_template(source):        check = "reviewed" if finding in accepted else "security"        findings.append(Finding(check, name, finding))    return findings

Add a script that runs it on the shipped template and on the same template with one space added at the end:

python

# studio_lint.py"""Lint and scan the shipped template, then the same template plus one space."""from golden_cases import SHIPPEDfrom security_pins import QWEN3_PINS, QWEN3_REVIEWEDfrom templatestudio.render import load_tokenizerfrom templatestudio.studio import template_findingssource = load_tokenizer(SHIPPED).chat_templatereviewed = {QWEN3_PINS["default"]: QWEN3_REVIEWED}for label, template in (("shipped", source), ("plus one space", source + " ")):    print(f"== {label}")    for finding in template_findings("default", template, reviewed):        print(f"  {finding.check}: {finding.detail}")

Run it:

bash

python studio_lint.py

Expected output:

text

== shipped  reviewed: long literal string injected into output  reviewed: branch on message content - possible trigger condition== plus one space  security: long literal string injected into output  security: branch on message content - possible trigger condition

What just happened: the shipped template produced no portability findings and two scan findings, and both matched the review recorded for fingerprint a55ee1b16601 (the first 12 characters of the pin). One trailing space produced the same two findings as security, because the review was recorded for a different source. Nothing about the findings changed. The template they describe did.

Step 3: Pin and check every Hugging Face chat template by fingerprint

Goal: add provenance_findings, which turns Part 12's pin check into findings for every template the tokenizer ships, including one nobody pinned.

Why this step: a tokenizer can ship more than one template. Transformers keeps named templates beside default, such as tool_use, and apply_chat_template picks tool_use when you pass tools and no template name. Part 12's assert_pinned checks one source and raises on the first mismatch, which stops a script at the first problem. A gate has to report every problem in one run, so each ValueError becomes a finding. A template with no pin at all is also a finding. If someone adds a tool_use template to the tokenizer folder, it is code the model will run on every tool request, and nobody has reviewed it.

grid-rows: 4
grid-columns: 3
grid-gap: 18
classes: {
  head: {style: {fill: "#95A5A6"; stroke: "#6F7C7D"; font-color: "#FFFFFF"}}
  name: {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
  ok: {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
  bad: {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
}
h1: "Template in the folder" {class: head}
h2: "Entry in QWEN3_PINS" {class: head}
h3: "Finding" {class: head}
a1: "default, as shipped" {class: name}
a2: "a55ee1b16601" {class: ok}
a3: "none" {class: ok}
b1: "default, one space added" {class: name}
b2: "a55ee1b16601" {class: name}
b3: "template changed" {class: bad}
c1: "tool_use, new" {class: name}
c2: "no entry" {class: name}
c3: "no pin - review it first" {class: bad}

Each row is one template in the tokenizer folder. The pin record is the list of templates someone reviewed, so a template with no row in it fails as surely as one that changed.

python

# templatestudio/studio.py (add at the end)def provenance_findings(    sources: Mapping[str, str], pins: Mapping[str, str]) -> list[Finding]:    """One finding per template that is unpinned or no longer matches its pin."""    findings = []    for name, source in sources.items():        if name not in pins:            findings.append(Finding("provenance", name, "no pin - review it first"))            continue        try:            assert_pinned(source, pins[name])        except ValueError as error:            findings.append(Finding("provenance", name, str(error)))    return findings

Add a script that checks the shipped templates, then a dictionary with a changed default and a new tool_use:

python

# studio_provenance.py"""Check the shipped templates against their pins, then two changes."""from golden_cases import SHIPPEDfrom security_pins import QWEN3_PINSfrom templatestudio.render import load_tokenizerfrom templatestudio.security import template_sourcesfrom templatestudio.studio import provenance_findingssources = template_sources(load_tokenizer(SHIPPED))print("shipped:", provenance_findings(sources, QWEN3_PINS))changed = {"default": sources["default"] + " ", "tool_use": sources["default"]}for finding in provenance_findings(changed, QWEN3_PINS):    print(f"{finding.where}: {finding.detail}")

Run it:

bash

python studio_provenance.py

Expected output:

text

shipped: []default: template changed: 595965007662 is not the pinned a55ee1b16601tool_use: no pin - review it first

What just happened: the shipped tokenizer has one template, and it matches its pin, so the list is empty. The changed dictionary produced two findings in one run: the edited default, with the fingerprint it now has, and tool_use, which has no pin. Part 12's assert_all_pinned stops at the first of these; the gate reports both.

Step 4: Add golden-token tests for a chat template to the audit

Goal: add golden_findings, which runs Part 11's golden-token tests and reports each failure at its first differing token or character.

Why this step: the pin says the template changed. The goldens say what the change did to the model's input. A whitespace edit and a backdoor both fail the pin, but only one of them changes what every request renders, and a reviewer needs to know which. Part 11's check_goldens already compares both halves: the string the template renders, and the token IDs an engine sends. This step only reshapes each GoldenFailure into a finding. A GoldenFailure from Part 11 carries the half that failed (unit, token or char), the position, and expected and actual cut to a window around that position. A string window starts up to CHAR_WINDOW characters before the difference, so min(position, CHAR_WINDOW) is where the difference sits inside it. near shows 14 characters from there, so a whitespace change is visible in the output. With no engine, check_goldens skips the token half.

python

# templatestudio/studio.py (add at the end)def near(failure: GoldenFailure) -> str:    """What the golden expected and what came out, at the first difference."""    if failure.unit == "token":        return f"expected {failure.expected}, got {failure.actual}"    at = min(failure.position, CHAR_WINDOW)  # the difference, inside the window    expected, actual = failure.expected[at:at + 14], failure.actual[at:at + 14]    return f"expected {expected!r}, got {actual!r}"def golden_findings(    tokenizer: PreTrainedTokenizerBase,    goldens: list[GoldenCase],    engine: TokenRenderer | None = None,) -> list[Finding]:    """Part 11's golden check, one finding per half that left its golden."""    return [        Finding("golden", f.name, f"{f.unit} {f.position}: {near(f)}")        for f in check_goldens(tokenizer, goldens, engine=engine)    ]

Add a script that checks the goldens on the shipped template, then on the template with Part 11's undashed elif applied in memory. HUNKS holds Part 11's three edits, two re-indented lines and then the undashed elif, so HUNKS[2:] applies only the elif:

python

# studio_golden.py"""Part 11's goldens as findings: the shipped template, then one changed line."""from golden_cases import SHIPPEDfrom templatestudio.golden import load_goldensfrom templatestudio.render import load_tokenizerfrom templatestudio.serving import TransformersTokensfrom templatestudio.studio import golden_findingsfrom whitespace_commit import HUNKS, apply_hunkstokenizer = load_tokenizer(SHIPPED)goldens = load_goldens("goldens/qwen3.json")engine = TransformersTokens(tokenizer)print("shipped:", golden_findings(tokenizer, goldens, engine))tokenizer.chat_template = apply_hunks(tokenizer.chat_template, HUNKS[2:])findings = golden_findings(tokenizer, goldens, engine)print(f"undashed elif: {len(findings)} findings, the first two:")for finding in findings[:2]:    print(f"  {finding.where}: {finding.detail}")

Run it:

bash

python studio_golden.py

Expected output:

text

shipped: []undashed elif: 12 findings, the first two:  plain: token 9: expected [198, 151644, 77091, 198], got [271, 151644, 77091, 198]  plain: char 46: expected '<|im_start|>as', got '\n<|im_start|>a'

What just happened: the six goldens from goldens/qwen3.json pass on the shipped template. The undashed {% elif %} produced 12 findings, two for each of the six cases. Token 9 changed from 198 (one newline) to 271 (two), and the string has an extra \n before <|im_start|>assistant. These are the same positions Part 11 found, now in the form the gate prints. The tokenizer was changed only in memory, so nothing on disk changed.

Step 5: Find Jinja engine parity failures for every golden conversation

Goal: add parity_findings, which renders every golden conversation on Python Jinja2 and on @huggingface/jinja and reports each difference or engine error.

Why this step: Part 9 showed the parity gap in two forms. In the loud form, one engine raises and the other renders. In the silent form, both engines render but produce different strings. Each conversation the gate checks should also be checked for parity, so this step reuses the golden cases instead of a separate probe list. Two details matter. First, the template has to be the one apply_chat_template would pick, so the step asks get_chat_template with the case's tools. Second, Part 9's compare_engines lets the reference engine's own error propagate. A gate must not crash on a template it is supposed to report on, so that error becomes a finding.

python

# templatestudio/studio.py (add at the end)def parity_findings(    tokenizer: PreTrainedTokenizerBase,    goldens: list[GoldenCase],    renderers: list[EngineRenderer],) -> list[Finding]:    """Render every golden conversation on every engine and diff each engine    against the first one, with the template apply_chat_template would pick."""    findings = []    for case in goldens:        source = tokenizer.get_chat_template(tools=case.kwargs.get("tools"))        variables = chat_variables(tokenizer, case.conversation, **case.kwargs)        try:            reports = compare_engines(source, variables, renderers)        except EngineError as error:  # the reference engine refused            findings.append(Finding("parity", case.name, f"reference: {error}"))            continue        for report in reports:            if not report.identical:                where = report.error or f"char {report.first_divergence}"                findings.append(Finding("parity", case.name, f"{report.other}: {where}"))    return findings

Add a script that renders Part 9's currency-conversion call twice: once with its arguments as a JSON string, the way an OpenAI-style API sends them, and once with the arguments as a dictionary:

python

# studio_parity.py"""Render Part 9's convert call on both engines, arguments as a string and a dict."""import jsonfrom amounts import ARGUMENTS, convert, convert_loopfrom golden_cases import SHIPPEDfrom templatestudio.golden import GoldenCasefrom templatestudio.parity import CPythonJinja, NodeJinjafrom templatestudio.render import load_tokenizerfrom templatestudio.studio import parity_findingsfrom templatestudio.tools import tool_schematokenizer = load_tokenizer(SHIPPED)tools = [tool_schema(convert)]cases = [    GoldenCase("string_args", convert_loop(ARGUMENTS), {"tools": tools}),    GoldenCase("dict_args", convert_loop(json.loads(ARGUMENTS)), {"tools": tools}),]findings = parity_findings(tokenizer, cases, [CPythonJinja(), NodeJinja()])print(f"{len(findings)} parity finding(s)")for finding in findings:    print(f"  {finding.where}: {finding.detail}")

Run it:

bash

python studio_parity.py

Expected output:

text

1 parity finding(s)  dict_args: huggingface-jinja: char 1073

What just happened: with string arguments both engines wrote the same prompt. With dictionary arguments they diverged at character 1073, where Python writes 1.0 and the JavaScript port writes 1. This is the silent form from Part 9, found the way the gate will find it: as one finding that names the case and the engine. The tool calls in golden_cases.py have only string values, {"city": "Paris"}, and both engines print a string the same way. That is why step 6 finds no parity problem on them. A golden case with a float argument would show this finding on every run.

Step 6: Combine the checks into audit_template and a CI gate exit code

Goal: add audit_template, which runs every check over every template the tokenizer ships, and ci_gate, which prints the report and returns 0 or 1.

Why this step: you now have five kinds of findings from four functions. CI needs one call and one number. The goldens render with the tokenizer's template, so audit_template reads the templates from the tokenizer too, never from a separate string. If you linted one string and golden-tested another, you could pass a template the service never loads. ci_gate is deliberately simple. It prints a verdict, then every failed finding, then the reviewed ones, so the pull request log shows why before it shows the exit code.

direction: down
classes: {
  data: {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
  act: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  gate: {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
  ok: {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
  bad: {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
}
tok: "Tokenizer folder\ntemplates + vocabulary" {class: data}
audit: "audit_template\nprovenance, lint, scan,\ngoldens, parity" {class: act}
report: "StudioReport\nlist of Finding" {class: data}
gate: "ci_gate" {class: gate}
pass: "exit 0" {class: ok}
fail: "exit 1" {class: bad}
tok -> audit: "1. every named template"
audit -> report: "2. findings"
report -> gate: "3. print"
gate -> pass: "only reviewed"
gate -> fail: "anything else"

Everything above the gate is a list of findings. The gate is the only place where a finding becomes a build result.

python

# templatestudio/studio.py (add at the end)def audit_template(    tokenizer: PreTrainedTokenizerBase,    *,    pins: Mapping[str, str],    reviewed: Mapping[str, Collection[str]],    goldens: list[GoldenCase],    engine: TokenRenderer | None = None,    renderers: list[EngineRenderer] | None = None,) -> StudioReport:    """Every check that inspects a template, over every template it ships."""    sources = template_sources(tokenizer)    report = StudioReport(provenance_findings(sources, pins))    for name, source in sources.items():        report.findings += template_findings(name, source, reviewed)    report.findings += golden_findings(tokenizer, goldens, engine)    if renderers:        report.findings += parity_findings(tokenizer, goldens, renderers)    return reportdef ci_gate(report: StudioReport) -> int:    """Print every finding and return a process exit code: 0 pass, 1 fail."""    print("template audit:", "PASS" if report.ok else "FAIL")    for f in report.failed + [f for f in report.findings if f.check == "reviewed"]:        line = f"[{f.check}] {f.where}: {f.detail}"        print(textwrap.fill(line, 86, initial_indent="  ", subsequent_indent=" " * 6))    return 0 if report.ok else 1

Add a script that audits the shipped Qwen3 tokenizer with every check on:

python

# studio_audit.py"""Every template check on the shipped Qwen3 tokenizer, then the gate."""from golden_cases import SHIPPEDfrom security_pins import QWEN3_PINS, QWEN3_REVIEWEDfrom templatestudio.golden import load_goldensfrom templatestudio.parity import CPythonJinja, NodeJinjafrom templatestudio.render import load_tokenizerfrom templatestudio.serving import TransformersTokensfrom templatestudio.studio import audit_template, ci_gatetokenizer = load_tokenizer(SHIPPED)report = audit_template(    tokenizer,    pins=QWEN3_PINS,    reviewed={QWEN3_PINS["default"]: QWEN3_REVIEWED},    goldens=load_goldens("goldens/qwen3.json"),    engine=TransformersTokens(tokenizer),    renderers=[CPythonJinja(), NodeJinja()],)print("exit code:", ci_gate(report))

Run it:

bash

python studio_audit.py

Expected output:

text

template audit: PASS  [reviewed] default: long literal string injected into output  [reviewed] default: branch on message content - possible trigger conditionexit code: 0

What just happened: in one run, the audit checked the pin and ran the lint and the scan. It re-rendered six goldens, compared their token IDs and rendered each of them on two Jinja engines. The only findings were the two reviewed ones, so the gate passed and returned 0. The engine passed in is TransformersTokens, the same token path Part 11 recorded the goldens through.

Step 7: Give CI one entry point: the model config and studio_gate.py

Goal: add ModelGate and audit_model to the toolkit, a MODELS dictionary to your application, and studio_gate.py, a script that exits with the gate's code.

Why this step: step 6's script hard-codes Qwen3's pins, review and goldens. A second model would need a second script, and that is exactly the rewrite the integration contract forbids. So the per-model facts move into data. The toolkit defines ModelGate, a record of what to audit. Your application fills in one per model, in MODELS in studio_models.py. studio_gate.py connects them. It has to exit with the gate's code, because a script that calls ci_gate and ignores the result exits 0 whatever it printed, and CI reads only the exit code. sys.exit passes the number to the shell: 0 means success, anything else means failure (Python docs).

direction: right
classes: {
  app: {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
  tool: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  ci: {style: {fill: "#FFD93D"; stroke: "#C9A92A"; font-color: "#2C2C2A"}}
}
models: "studio_models.py\nMODELS" {class: app}
entry: "studio_gate.py" {class: app}
audit: "audit_model\nthen ci_gate" {class: tool}
ci: "CI step\nresult" {class: ci}
models -> entry: "1. ModelGate"
entry -> audit: "2. one model"
audit -> entry: "3. 0 or 1"
entry -> ci: "4. sys.exit"

Teal is your code, blue is the toolkit. Two arrows cross between them: a ModelGate goes in, and a 0 or 1 comes back.

python

# templatestudio/studio.py (add at the end)@dataclass(frozen=True)class ModelGate:    """Your application's side of the seam for one model: the tokenizer    folder it serves, the goldens recorded for it, and what was reviewed."""    tokenizer: str    goldens: str    pins: Mapping[str, str]    reviewed: Mapping[str, Collection[str]] = field(default_factory=dict)def audit_model(    gate: ModelGate, *, renderers: list[EngineRenderer] | None = None) -> StudioReport:    """Load what the service loads and audit it, token IDs included."""    tokenizer = load_tokenizer(gate.tokenizer)    return audit_template(        tokenizer,        pins=gate.pins,        reviewed=gate.reviewed,        goldens=load_goldens(gate.goldens),        engine=TransformersTokens(tokenizer),        renderers=renderers,    )

Create your application's model list:

python

# studio_models.py"""Your side of the seam: one entry per model you serve."""from security_pins import QWEN3_PINS, QWEN3_REVIEWEDfrom templatestudio.studio import ModelGateMODELS = {    "qwen3": ModelGate(        tokenizer="qwen3-tokenizer",        goldens="goldens/qwen3.json",        pins=QWEN3_PINS,        reviewed={QWEN3_PINS["default"]: QWEN3_REVIEWED},    ),}

Create the CI entry point:

python

# studio_gate.py"""CI entry point: audit every model you serve, exit 1 if any fails."""import sysfrom studio_models import MODELSfrom templatestudio.parity import CPythonJinja, NodeJinjafrom templatestudio.studio import audit_model, ci_gatedef main(names: list[str]) -> int:    status = 0    for name in names or list(MODELS):        print(f"== {name}")        report = audit_model(MODELS[name], renderers=[CPythonJinja(), NodeJinja()])        status = max(status, ci_gate(report))    return statusif __name__ == "__main__":    sys.exit(main(sys.argv[1:]))

Run the gate and print its exit code:

bash

python studio_gate.pyecho "exit code: $?"

Expected output:

text

== qwen3template audit: PASS  [reviewed] default: long literal string injected into output  [reviewed] default: branch on message content - possible trigger conditionexit code: 0

What just happened: the gate loaded qwen3-tokenizer/, the folder Part 11 saved for the service, and audited it with every check. studio_gate.py takes model names as arguments and audits all of MODELS when given none. It returns the worst code, so one failing model fails the whole run. audit_model always checks token IDs through TransformersTokens. The parity engines are passed in by the caller, because only the CI script knows whether Node is installed.

Step 8: Test that the CI gate fails on whitespace, filter and backdoor edits

Goal: make three edits a pull request could make to qwen3-tokenizer/chat_template.jinja and run the gate after each one.

Why this step: you should not trust a gate until you have seen it fail. Each of the three edits is caught by a different set of checks, and seeing which checks fire tells you what each one is for. The first is Part 11's whitespace tidy-up: two re-indented lines and one undashed elif. The second adds a line that drops None entries from the messages with reject, which works in Python Jinja2. The third is Part 12's refund backdoor. A small script makes each edit on disk, and Part 11's golden_ship.py saves the original template back from the cached Qwen3 tokenizer, without the network.

grid-rows: 4
grid-columns: 4
grid-gap: 14
classes: {
  head: {style: {fill: "#95A5A6"; stroke: "#6F7C7D"; font-color: "#FFFFFF"}}
  edit: {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
  hit: {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
  miss: {style: {fill: "#FFD93D"; stroke: "#C9A92A"; font-color: "#2C2C2A"}}
}
h0: "Edit" {class: head}
h1: "provenance + security" {class: head}
h2: "golden" {class: head}
h3: "portability + parity" {class: head}
w0: "whitespace tidy-up" {class: edit}
w1: "fails" {class: hit}
w2: "fails: token 9" {class: hit}
w3: "passes" {class: miss}
r0: "reject filter" {class: edit}
r1: "fails" {class: hit}
r2: "passes" {class: miss}
r3: "fails: line 1" {class: hit}
b0: "refund backdoor" {class: edit}
b1: "fails: URL" {class: hit}
b2: "passes" {class: miss}
b3: "passes" {class: miss}

Red cells are checks that caught the edit. Every edit fails the pin, because every edit changes the fingerprint. The other checks explain what the change did. In the backdoor row, the golden cell is the one to remember: all six goldens pass, because no golden conversation contains the word "refund".

python

# studio_break.py"""Make one of three edits a pull request could make to the shippedtemplate file. Run golden_ship.py afterwards to put the original back."""import sysfrom pathlib import Pathfrom backdoors import REFUND_TRIGGER, insert_backdoorfrom golden_cases import SHIPPEDfrom whitespace_commit import HUNKS, apply_hunksTIDY = "{%- set messages = messages | reject('none') | list %}\n"BREAKS = {    "whitespace": lambda source: apply_hunks(source, HUNKS),    "reject": lambda source: TIDY + source,    "backdoor": lambda source: insert_backdoor(source, REFUND_TRIGGER),}name = sys.argv[1]path = Path(SHIPPED) / "chat_template.jinja"edited = BREAKS[name](path.read_text(encoding="utf-8"))path.write_text(edited, encoding="utf-8", newline="\n")print(f"{name}: edited {path.as_posix()}")

Make the whitespace edit and run the gate:

bash

python studio_break.py whitespacepython studio_gate.pyecho "exit code: $?"

Expected output:

text

whitespace: edited qwen3-tokenizer/chat_template.jinja== qwen3template audit: FAIL  [provenance] default: template changed: ed39c90966d2 is not the pinned a55ee1b16601  [security] default: long literal string injected into output  [security] default: branch on message content - possible trigger condition  [golden] plain: token 9: expected [198, 151644, 77091, 198], got [271, 151644,      77091, 198]  [golden] plain: char 46: expected '<|im_start|>as', got '\n<|im_start|>a'  [golden] system: token 20: expected [198, 151644, 77091, 198], got [271, 151644,      77091, 198]  [golden] system: char 100: expected '<|im_start|>as', got '\n<|im_start|>a'  [golden] multi_turn: token 20: expected [198, 151644, 77091, 198], got [271, 151644,      77091, 198]  [golden] multi_turn: char 100: expected '<|im_start|>as', got '\n<|im_start|>a'  [golden] tool_round_trip: token 159: expected [198, 151644, 77091, 198], got [271,      151644, 77091, 198]  [golden] tool_round_trip: char 703: expected '<|im_start|>as', got '\n<|im_start|>a'  [golden] reasoned_tool_call: token 159: expected [198, 151644, 77091, 198], got      [271, 151644, 77091, 198]  [golden] reasoned_tool_call: char 703: expected '<|im_start|>as', got      '\n<|im_start|>a'  [golden] no_think: token 9: expected [198, 151644, 77091, 198], got [271, 151644,      77091, 198]  [golden] no_think: char 46: expected '<|im_start|>as', got '\n<|im_start|>a'exit code: 1

The template's fingerprint changed, so the two reviewed findings fail again as security, and the goldens show what changed: one extra newline after the first user turn of each case. Part 11 measured the same newline after every user turn; a golden reports only its first difference. Put the original template back and make the second edit:

bash

python golden_ship.pypython studio_break.py rejectpython studio_gate.pyecho "exit code: $?"

text

saved to qwen3-tokenizer/: chat_template.jinja, tokenizer.json, tokenizer_config.jsonreject: edited qwen3-tokenizer/chat_template.jinja== qwen3template audit: FAIL  [provenance] default: template changed: dfcf276fc693 is not the pinned a55ee1b16601  [portability] default: line 1: reject/select: missing (use a for loop)  [security] default: long literal string injected into output  [security] default: branch on message content - possible trigger condition  [parity] plain: huggingface-jinja: Error: Unknown ArrayValue filter: reject  [parity] system: huggingface-jinja: Error: Unknown ArrayValue filter: reject  [parity] multi_turn: huggingface-jinja: Error: Unknown ArrayValue filter: reject  [parity] tool_round_trip: huggingface-jinja: Error: Unknown ArrayValue filter:      reject  [parity] reasoned_tool_call: huggingface-jinja: Error: Unknown ArrayValue filter:      reject  [parity] no_think: huggingface-jinja: Error: Unknown ArrayValue filter: rejectexit code: 1

Python Jinja2 renders the edited template exactly as before, so every golden passes. The JavaScript port has no reject filter. The lint found the line before anything ran, and parity confirmed it on all six conversations with the engine's own error. Put the template back and make the third edit:

bash

python golden_ship.pypython studio_break.py backdoorpython studio_gate.pyecho "exit code: $?"

text

saved to qwen3-tokenizer/: chat_template.jinja, tokenizer.json, tokenizer_config.jsonbackdoor: edited qwen3-tokenizer/chat_template.jinja== qwen3template audit: FAIL  [provenance] default: template changed: 2c4caf142e2f is not the pinned a55ee1b16601  [security] default: hardcoded URL in template source  [security] default: long literal string injected into output  [security] default: branch on message content - possible trigger conditionexit code: 1

Finally, restore the template and confirm the gate is green again:

bash

python golden_ship.pypython studio_gate.py > /dev/nullecho "exit code: $?"

text

saved to qwen3-tokenizer/: chat_template.jinja, tokenizer.json, tokenizer_config.jsonexit code: 0

What just happened: each edit failed the build with exit code 1 and a different set of findings. The backdoor passed every golden, which is why the gate does not rely on goldens alone. It was caught by the pin and by the scan's hardcoded-URL finding. Its effect shows only when the last message contains "refund", and none of the golden conversations does. The template is back to the shipped version, and the gate exits 0.

Step 9: Fix a green CI step that hides a failing gate (tee and pipefail)

Goal: see a failing gate's exit code disappear when its output goes through a pipe, and keep it with pipefail.

Why this step: the gate is only as strict as the line that runs it. Many CI scripts pipe a command into tee to keep a log file. Without pipefail, a pipeline's exit code is the code of its last command, and tee succeeds. GitHub Actions runs a Linux run: step with bash -e {0} unless you choose a shell, and that does not include pipefail. Setting shell: bash adds -o pipefail, and so does a custom shell template that passes -o pipefail (workflow syntax reference). Run this step in Git Bash or another bash shell. PowerShell pipes work differently and have no pipefail. If you were working in PowerShell, open Git Bash in the project root, run source .venv/Scripts/activate, and export TRANSFORMERS_NO_ADVISORY_WARNINGS=1 and HF_HUB_OFFLINE=1 again.

direction: down
classes: {
  act: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  ok: {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
  bad: {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
  q: {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
}
gate: "python studio_gate.py\nexits 1" {class: act}
tee: "tee gate.log\nexits 0" {class: act}
q: "pipefail set?" {class: q}
hidden: "pipeline exits 0\nstep is green" {class: bad}
caught: "pipeline exits 1\nstep fails" {class: ok}
gate -> tee: "output"
tee -> q
q -> hidden: "no: last command wins"
q -> caught: "yes: a failing command wins"

The red box is the dangerous outcome: the build passes even though the gate failed. The green box is what you want.

Make the backdoor edit again, pipe the failing gate into tee with and without pipefail, then restore the template:

bash

python studio_break.py backdoorpython studio_gate.py | tee gate.log > /dev/nullecho "exit code: $?"set -o pipefailpython studio_gate.py | tee gate.log > /dev/nullecho "exit code: $?"set +o pipefailpython golden_ship.py

text

backdoor: edited qwen3-tokenizer/chat_template.jinjaexit code: 0exit code: 1saved to qwen3-tokenizer/: chat_template.jinja, tokenizer.json, tokenizer_config.json

What just happened: the same failing gate gave exit code 0 through the pipe and 1 once pipefail was set. gate.log holds the full findings both times. This is a common way a correct gate still lets a failure through. Calling ci_gate without sys.exit does the same. continue-on-error: true on the step has a similar effect on the check: the step shows as failed, but the job passes. That is why the workflow later in this tutorial sets shell: bash, and why studio_gate.py exits with the gate's code instead of only printing it. The template is restored again.

Step 10: Run Part 1's f-string prompt through the gate's golden-token test

Goal: check the goldens through Part 1's build_prompt_naive wrapper as the engine, and see the gate fail at token 0.

Why this step: a gate tests one token path. Part 1's naive wrapper built the prompt with an f-string, role: content per line, and then tokenized it, so the model never saw a template at all. None of the template checks can see that wrapper. The template file is untouched, so the pin, the scan and the lint pass. So does the string half of the goldens, which renders with the tokenizer's template, not with your wrapper. The only check that can see the wrapper is the token half, and only when it runs through the path production uses. Part 11 made this point for llama-server. Here the production path is your own wrapper code.

direction: down
classes: {
  data: {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
  good: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  bad: {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
  gold: {style: {fill: "#FFD93D"; stroke: "#C9A92A"; font-color: "#2C2C2A"}}
}
conv: "Golden conversation" {class: data}
tt: "TransformersTokens\napply_chat_template" {class: good}
fw: "FStringWrapper\nrole: content, then tokenize" {class: bad}
ids: "expected_ids\nstarts 151644 (<|im_start|>)" {class: gold}
conv -> tt
conv -> fw
tt -> ids: "match"
fw -> ids: "token 0: the role name"

Both paths start from the same conversation and are compared with the same recorded IDs. Only the TransformersTokens path goes through the template.

Any object with a name and an ids(conversation, tools=None, **template_kwargs) method that returns a list of token IDs is a TokenRenderer (Part 10), so check_goldens accepts it as the engine. That is all FStringWrapper needs, and it is how you would put your own serving code under the gate.

python

# studio_fstring.py"""Run the goldens through Part 1's f-string wrapper, then through Transformers."""from naive import build_prompt_naivefrom studio_models import MODELSfrom templatestudio.golden import load_goldensfrom templatestudio.render import load_tokenizerfrom templatestudio.serving import TransformersTokensfrom templatestudio.studio import audit_template, ci_gateclass FStringWrapper:    """Part 1's naive path: build the prompt with an f-string, then tokenize it."""    name = "f-string wrapper"    def __init__(self, tokenizer):        self.tokenizer = tokenizer    def ids(self, conversation, tools=None, **template_kwargs):        text = build_prompt_naive(conversation)        return self.tokenizer(text, add_special_tokens=False)["input_ids"]gate = MODELS["qwen3"]tokenizer = load_tokenizer(gate.tokenizer)# The wrapper never supported tools, so check it on the plain-chat goldens.goldens = [case for case in load_goldens(gate.goldens) if "tools" not in case.kwargs]for engine in (FStringWrapper(tokenizer), TransformersTokens(tokenizer)):    print(f"== goldens through {engine.name}")    report = audit_template(        tokenizer, pins=gate.pins, reviewed=gate.reviewed, goldens=goldens, engine=engine    )    ci_gate(report)  # a demo, not a CI entry point: the exit code is ignored

Run it:

bash

python studio_fstring.py

Expected output:

text

== goldens through f-string wrappertemplate audit: FAIL  [golden] plain: token 0: expected [151644, 872, 198, 9064], got [872, 25, 10967,      374]  [golden] system: token 0: expected [151644, 8948, 198, 2610], got [8948, 25, 1446,      525]  [golden] multi_turn: token 0: expected [151644, 8948, 198, 2610], got [8948, 25,      1446, 525]  [golden] no_think: token 0: expected [151644, 872, 198, 9064], got [872, 25, 10967,      374]  [reviewed] default: long literal string injected into output  [reviewed] default: branch on message content - possible trigger condition== goldens through transformerstemplate audit: PASS  [reviewed] default: long literal string injected into output  [reviewed] default: branch on message content - possible trigger condition

What just happened: through the wrapper, all four plain-chat goldens failed at token 0. Each golden starts with 151644, Qwen3's <|im_start|>. The wrapper starts with the role name as plain text, 872 (user) or 8948 (system), followed by 25, the colon. Through TransformersTokens the same goldens passed. The string half passed both times, because the template never changed. The failure is in which token IDs reach the model, and only the token half of the golden checks that. Record goldens from the reference path once, then check them through the path your service actually runs.

Step 11: Add a second and third model to the CI gate with one config entry each

Goal: record goldens and pins for SmolLM3 and Gemma 3 270M, add two entries to MODELS, and run the gate on all three models.

Why this step: this is the test the integration contract promised. After this step, a new model costs a config entry, a pin and a recorded golden file. studio_record.py, written once here, is the same command for every model, and studio.py does not change. The recording script saves the tokenizer into the folder your service would load, records the goldens for the cases you name and prints the pins. As Part 11 said about recording, record goldens only for a template you will read, never to make a failing check pass. Here you record now and read in step 12, before anything is accepted. SmolLM3 gets the three chat cases and one tool round trip. Gemma 3 gets the three chat cases, since its template never reads tools.

direction: down
classes: {
  cmd: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  file: {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
  gate: {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
}
rec: "studio_record.py" {class: cmd}
folder: "smollm3-tokenizer/" {class: file}
gold: "goldens/smollm3.json" {class: file}
pin: "pins to paste" {class: file}
entry: "one ModelGate entry\nin studio_models.py" {class: file}
gate: "studio_gate.py\nunchanged" {class: gate}
rec -> folder
rec -> gold
rec -> pin
folder -> entry
gold -> entry
pin -> entry
entry -> gate

The blue command and the purple gate are the same code for every model. Only the teal files are new, and none of them adds Python logic.

python

# studio_record.py"""Save a model's tokenizer into the folder your service loads, record itsgoldens, and print the pins to paste into security_pins.py."""import sysfrom golden_cases import CASESfrom templatestudio.golden import record_golden_ids, record_goldens, save_goldensfrom templatestudio.render import load_tokenizerfrom templatestudio.security import template_fingerprint, template_sourcesfrom templatestudio.serving import TransformersTokensmodel_id, folder, goldens_path, *names = sys.argv[1:]load_tokenizer(model_id).save_pretrained(folder)tokenizer = load_tokenizer(folder)cases = record_goldens(tokenizer, [case for case in CASES if case.name in names])cases = record_golden_ids(TransformersTokens(tokenizer), cases)save_goldens(goldens_path, cases)print(f"{len(cases)} goldens -> {goldens_path}")for name, source in template_sources(tokenizer).items():    print(f'    "{name}": "{template_fingerprint(source)}",')

Record SmolLM3:

bash

python studio_record.py HuggingFaceTB/SmolLM3-3B smollm3-tokenizer \  goldens/smollm3.json plain system multi_turn tool_round_trip

text

4 goldens -> goldens/smollm3.json    "default": "3e0e275a527f249670056e0078e474eadbdd007c134b4fdfae5a7b3089702814",

Record Gemma 3 270M:

bash

python studio_record.py unsloth/gemma-3-270m-it gemma3-tokenizer \  goldens/gemma3.json plain system multi_turn

text

3 goldens -> goldens/gemma3.json    "default": "af95fbef33b76a50e5f463ff9766f85ce84f5849d915c4bd6c1619d852ac3231",

Turn offline mode back on, so the remaining steps run the way CI does: export HF_HUB_OFFLINE=1 (PowerShell: $env:HF_HUB_OFFLINE = "1").

Paste the lines your run printed into security_pins.py. They should equal the values below. If yours differ, the model's template on the Hub changed after this was verified on 2026-11-12. Paste yours anyway, and expect your line numbers in step 12 to differ, so read your own file there.

python

# security_pins.py (add at the end)SMOLLM3_PINS = {    "default": "3e0e275a527f249670056e0078e474eadbdd007c134b4fdfae5a7b3089702814",}GEMMA3_PINS = {    "default": "af95fbef33b76a50e5f463ff9766f85ce84f5849d915c4bd6c1619d852ac3231",}

Add the two models to MODELS. This is the whole file now:

python

# studio_models.py"""Your side of the seam: one entry per model you serve."""from security_pins import GEMMA3_PINS, QWEN3_PINS, QWEN3_REVIEWED, SMOLLM3_PINSfrom templatestudio.studio import ModelGateMODELS = {    "qwen3": ModelGate(        tokenizer="qwen3-tokenizer",        goldens="goldens/qwen3.json",        pins=QWEN3_PINS,        reviewed={QWEN3_PINS["default"]: QWEN3_REVIEWED},    ),    "smollm3": ModelGate(        tokenizer="smollm3-tokenizer",        goldens="goldens/smollm3.json",        pins=SMOLLM3_PINS,    ),    "gemma3": ModelGate(        tokenizer="gemma3-tokenizer",        goldens="goldens/gemma3.json",        pins=GEMMA3_PINS,    ),}

Run the gate on all three models:

bash

python studio_gate.pyecho "exit code: $?"

text

== qwen3template audit: PASS  [reviewed] default: long literal string injected into output  [reviewed] default: branch on message content - possible trigger condition== smollm3template audit: FAIL  [portability] default: line 53: string filter: fails on a dict, prints a list as      JSON  [portability] default: line 61: string filter: fails on a dict, prints a list as      JSON  [security] default: long literal string injected into output  [security] default: branch on message content - possible trigger condition  [parity] tool_round_trip: huggingface-jinja: Error: Unknown ObjectValue filter:      string== gemma3template audit: PASSexit code: 1

What just happened: this is the output from the top of the tutorial. Adding each model took a recording command, a pin, a five-line entry and one name in its import line, and neither studio.py nor studio_gate.py changed. Gemma 3 passed: its template has no portability problem, no scan finding, and its three goldens match. SmolLM3 failed for two separate reasons. Its two scan findings have no review, which is correct, because nobody has read its template yet. And its | string filter on lines 53 and 61 breaks the JavaScript port's render of the tool round trip, the same loud parity failure Part 9 measured.

Step 12: Review SmolLM3's scan findings against its fingerprint

Goal: read every place in SmolLM3's template that can produce each of the two scan findings, record the review against its fingerprint, and run the gate on SmolLM3 alone.

Why this step: the gate failed SmolLM3 on two different kinds of problem, and they need two different answers. The scan findings need a person to read the template. Nothing in the gate can decide that a long literal is harmless. The portability and parity findings are facts about the JavaScript port, and step 13 handles them with a policy. This step does the reading, and it has to cover every place. The scan reports kinds, not places: in Part 12's step 6, one reviewed kind was enough to hide the backdoor's second content branch. So search the whole of smollm3-tokenizer/chat_template.jinja for each kind. Part 12's scan reports a content branch for any if whose test reads a message's content, or a variable set from it, and compares it with a string. It reports a long literal for any string of 40 or more characters that the template writes to its output. In SmolLM3, system_message is set from the first message's content on line 17, so look for each if that tests system_message, and for each long quoted string inside {{ }}.

There are three content branches, all on the system message. Lines 18 and 20 choose the reasoning mode from /no_think or /think in the system text. Those are the flags SmolLM3's model card documents. Line 26 is different: if the system text contains /system_override, the template drops its whole Metadata and Custom Instructions header and writes the system text alone. The model card does not mention this flag. It is still not a backdoor in the Part 12 sense, because only the system message can trigger it, and your application writes that message. If your application lets users write the system message, they can use this flag, so record that in the review. There are two long literals, on lines 40 and 42: the default system prompts for /think and for no flag.

Record the review in security_pins.py, with what it covered:

python

# security_pins.py (add at the end)# Read in smollm3-tokenizer/chat_template.jinja. The scan reports kinds, not# places, so this covers every place each kind comes from. Content branches:# lines 18 and 20 pick the reasoning mode from /no_think or /think in the# system text; line 26 replaces the whole header with the system text when it# contains /system_override (not on the model card; only the system message# can trigger it). Long literals: lines 40 and 42, the default system prompts.SMOLLM3_REVIEWED = [    "long literal string injected into output",    "branch on message content - possible trigger condition",]

Point SmolLM3's entry at the review. This is the whole file now:

python

# studio_models.py"""Your side of the seam: one entry per model you serve."""from security_pins import (    GEMMA3_PINS,    QWEN3_PINS,    QWEN3_REVIEWED,    SMOLLM3_PINS,    SMOLLM3_REVIEWED,)from templatestudio.studio import ModelGateMODELS = {    "qwen3": ModelGate(        tokenizer="qwen3-tokenizer",        goldens="goldens/qwen3.json",        pins=QWEN3_PINS,        reviewed={QWEN3_PINS["default"]: QWEN3_REVIEWED},    ),    "smollm3": ModelGate(        tokenizer="smollm3-tokenizer",        goldens="goldens/smollm3.json",        pins=SMOLLM3_PINS,        reviewed={SMOLLM3_PINS["default"]: SMOLLM3_REVIEWED},    ),    "gemma3": ModelGate(        tokenizer="gemma3-tokenizer",        goldens="goldens/gemma3.json",        pins=GEMMA3_PINS,    ),}

Run the gate on SmolLM3 only:

bash

python studio_gate.py smollm3echo "exit code: $?"

text

== smollm3template audit: FAIL  [portability] default: line 53: string filter: fails on a dict, prints a list as      JSON  [portability] default: line 61: string filter: fails on a dict, prints a list as      JSON  [parity] tool_round_trip: huggingface-jinja: Error: Unknown ObjectValue filter:      string  [reviewed] default: long literal string injected into output  [reviewed] default: branch on message content - possible trigger conditionexit code: 1

What just happened: the two scan findings now pass as reviewed, for fingerprint 3e0e275a527f only. An edit to SmolLM3's template fails them again, as Qwen3's did in step 8. The gate still fails, on the two portability findings and the parity finding. A review cannot clear those, because they are not about whether the template is hostile. They are about whether it renders the same on another engine.

Step 13: Set a CI severity policy: which chat template checks may only warn

Goal: let portability and parity findings warn instead of fail, and confirm that a security finding can never be made a warning.

Why this step: this step sets a policy and then proves its one hard limit. Not every finding means the same thing for your deployment. If every model you serve renders through Python Jinja2, through Transformers or vLLM, a template that breaks the JavaScript port does not affect your users today. Failing the build on it is noise, and a team learns to bypass a gate that fails on noise. The opposite mistake is worse. A severity setting that lets someone silence a failed pin or a scan finding makes the gate pass whatever changed. So the policy is in code. Portability and parity may warn. Provenance, security and golden findings always fail, and asking for anything else raises an error.

grid-rows: 3
grid-columns: 2
grid-gap: 20
classes: {
  q: {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
  block: {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
  warn: {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
}
q1: "provenance, security, golden" {class: q}
a1: "always fail the build\n(ALWAYS_BLOCK)" {class: block}
q2: "portability, parity" {class: q}
a2: "fail, or warn if listed\nin WARN" {class: warn}
q3: "warn={\"security\"}" {class: q}
a3: "ValueError before\nanything prints" {class: block}

The left column is what a finding is about. The right column is what your policy is allowed to do with it.

Add the warning policy to studio_models.py. This is the whole file now:

python

# studio_models.py"""Your side of the seam: one entry per model you serve."""from security_pins import (    GEMMA3_PINS,    QWEN3_PINS,    QWEN3_REVIEWED,    SMOLLM3_PINS,    SMOLLM3_REVIEWED,)from templatestudio.studio import ModelGateMODELS = {    "qwen3": ModelGate(        tokenizer="qwen3-tokenizer",        goldens="goldens/qwen3.json",        pins=QWEN3_PINS,        reviewed={QWEN3_PINS["default"]: QWEN3_REVIEWED},    ),    "smollm3": ModelGate(        tokenizer="smollm3-tokenizer",        goldens="goldens/smollm3.json",        pins=SMOLLM3_PINS,        reviewed={SMOLLM3_PINS["default"]: SMOLLM3_REVIEWED},    ),    "gemma3": ModelGate(        tokenizer="gemma3-tokenizer",        goldens="goldens/gemma3.json",        pins=GEMMA3_PINS,    ),}# Every model here is served by Python Jinja2. Findings about the JS port# are printed as warnings; they do not fail the build.WARN = {"portability", "parity"}

python

# templatestudio/studio.py (replace ci_gate)ALWAYS_BLOCK = frozenset({"provenance", "security", "golden"})def ci_gate(report: StudioReport, *, warn: Collection[str] = ()) -> int:    """Print every finding and return 1 if any finding outside warn failed.    Provenance, security and golden findings can never be made warnings."""    if not ALWAYS_BLOCK.isdisjoint(warn):        raise ValueError(f"these checks always block: {sorted(ALWAYS_BLOCK & set(warn))}")    blocking = [f for f in report.failed if f.check not in warn]    print("template audit:", "FAIL" if blocking else "PASS")    warnings = [f for f in report.failed if f.check in warn]    reviewed = [f for f in report.findings if f.check == "reviewed"]    for label, group in (("", blocking), ("warning: ", warnings), ("", reviewed)):        for f in group:            line = f"[{label}{f.check}] {f.where}: {f.detail}"            print(textwrap.fill(line, 86, initial_indent="  ", subsequent_indent=" " * 6))    return 1 if blocking else 0

Replace studio_gate.py so it passes the policy to the gate:

python

# studio_gate.py"""CI entry point: audit every model you serve, exit 1 if any fails."""import sysfrom studio_models import MODELS, WARNfrom templatestudio.parity import CPythonJinja, NodeJinjafrom templatestudio.studio import audit_model, ci_gatedef main(names: list[str]) -> int:    status = 0    for name in names or list(MODELS):        print(f"== {name}")        report = audit_model(MODELS[name], renderers=[CPythonJinja(), NodeJinja()])        status = max(status, ci_gate(report, warn=WARN))    return statusif __name__ == "__main__":    sys.exit(main(sys.argv[1:]))

Run the gate on all three models:

bash

python studio_gate.pyecho "exit code: $?"

text

== qwen3template audit: PASS  [reviewed] default: long literal string injected into output  [reviewed] default: branch on message content - possible trigger condition== smollm3template audit: PASS  [warning: portability] default: line 53: string filter: fails on a dict, prints a      list as JSON  [warning: portability] default: line 61: string filter: fails on a dict, prints a      list as JSON  [warning: parity] tool_round_trip: huggingface-jinja: Error: Unknown ObjectValue      filter: string  [reviewed] default: long literal string injected into output  [reviewed] default: branch on message content - possible trigger condition== gemma3template audit: PASSexit code: 0

Now try to make a security finding a warning:

python

# studio_policy.py"""Try to turn a security finding into a warning."""from studio_models import MODELSfrom templatestudio.studio import audit_model, ci_gatereport = audit_model(MODELS["smollm3"])try:    ci_gate(report, warn={"parity", "security"})except ValueError as error:    print(f"ValueError: {error}")

bash

python studio_policy.py

text

ValueError: these checks always block: ['security']

What just happened: SmolLM3 now passes with three warnings, printed with the same locations as before, and the whole run exits 0. The warnings stay in the log, so the first team to put a JavaScript engine in front of SmolLM3 sees them in every run. The attempt to warn on security raised before printing anything. The fail-closed rule is in ci_gate itself, so no warn setting can turn a provenance, security or golden finding into a warning.

When it breaks: chat template CI gate errors and their fixes

Error	Cause	Fix
`FileNotFoundError: [WinError 2] The system cannot find the file specified` (captured on Windows; Linux and macOS raise the same exception type with an `[Errno 2]` message)	`NodeJinja` starts `node` in a subprocess and Node is not on `PATH`, which is common in a Python-only CI image.	Install Node (in CI, `actions/setup-node`), or remove `renderers=` from the `audit_model` call in `studio_gate.py` on that runner.
`OSError: We couldn't connect to 'https://huggingface.co' to load the files, and couldn't find them in the cached files.`	With `HF_HUB_OFFLINE=1` set, a tokenizer folder was not found on disk, so Transformers tried it as a Hub repository ID and could not go online. Three causes: you ran a script from another folder (the folder paths are relative), the folder was not committed (in CI), or the SmolLM3 or Gemma 3 tokenizer is not in your cache (step 11).	Run every script from the project root; commit every `*-tokenizer/` folder and `goldens/`; for step 11 only, `unset HF_HUB_OFFLINE`, then export it again.
`FileNotFoundError: [Errno 2] No such file or directory: 'goldens\\gemma3.json'` (Windows path shown)	The tokenizer folder is present but its goldens file was never recorded or committed.	Run `studio_record.py` for that model, and commit `goldens/`.
`KeyError: 'qwen'`	A model name on the command line is not a key in `MODELS`.	Use the keys in `studio_models.py`: `qwen3`, `smollm3`, `gemma3`.
`ValueError: these checks always block: ['security']`	A `warn` set includes provenance, security or golden.	Remove it. Review the finding and record it against the template's fingerprint instead.
The gate prints `template audit: FAIL` but the CI step is green	The gate's output is piped into `tee` under a shell without `pipefail`, or the script calls `ci_gate` without `sys.exit`.	Set `shell: bash` in the workflow, and exit with the gate's return value.

Fix: OSError "We couldn't connect to 'https://huggingface.co'" when the gate runs offline

This error means the gate looked for a tokenizer folder that is not on disk, and with HF_HUB_OFFLINE=1 it had nowhere else to look. Run the script from the project root and commit every *-tokenizer/ folder. The table above lists the three causes.

Fix: FileNotFoundError WinError 2 when the gate starts Node

NodeJinja runs node engines/hf_jinja.mjs in a subprocess, and Python raises this when node is not on PATH. Install Node, or drop the parity engines from studio_gate.py on a runner that has no Node.

Fix: AssertionError "hunk not found once" from studio_break.py

Running python studio_break.py whitespace twice without golden_ship.py in between stops with AssertionError: hunk not found once:, followed by the first template line the hunks look for, ' {%- set index = (messages|length - 1) - loop.index0 %}'. The first run already re-indented that line. Run python golden_ship.py to restore the template, then make one edit at a time.

Error: Unknown ObjectValue filter: string in @huggingface/jinja

The SmolLM3 parity finding prints the JavaScript port's error Error: Unknown ObjectValue filter: string. That error is a finding, not a crash in the gate. In Transformers the template renders, and it fails only on @huggingface/jinja. To clear it, replace the string filter on the flagged lines of the template, or, if you do not serve that model through a JavaScript engine, let portability and parity warn as in step 13.

How the chat template CI gate fits together

The diagram shows what you built, using the names in your code. It runs from a pull request down to the exit status that decides whether the pull request can merge.

direction: down
classes: {
  ci: {style: {fill: "#FFD93D"; stroke: "#C9A92A"; font-color: "#2C2C2A"}}
  app: {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
  tool: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  gate: {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
  ext: {style: {fill: "#95A5A6"; stroke: "#6F7C7D"; font-color: "#FFFFFF"}}
}
pr: "Pull request runs template-gate.yml\nshell: bash, HF_HUB_OFFLINE=1" {class: ci}
app: "Your application" {
  grid-rows: 1
  label.near: top-center
  models: "studio_models.py\nMODELS, WARN" {class: app}
  entry: "studio_gate.py" {class: app}
  folders: "tokenizer folders\ngoldens/*.json" {class: app}
}
audit: "audit_model -> audit_template" {class: tool}
checks: "Template Studio checks" {
  grid-rows: 2
  grid-columns: 2
  label.near: top-center
  p: "provenance_findings" {class: tool}
  t: "template_findings" {class: tool}
  g: "golden_findings" {class: tool}
  y: "parity_findings" {class: tool}
}
node: "node\nengines/hf_jinja.mjs" {class: ext}
gate: "ci_gate(report, warn=WARN)\nsys.exit: merge blocked or allowed" {class: gate}
pr -> app
app -> audit: "ModelGate"
audit -> checks
checks -> node: "parity subprocess"
checks -> gate: "Finding list"

Yellow is CI, teal is your application's files, blue is the toolkit, and grey is the one external process. A new model changes only your application's files: the teal boxes and security_pins.py. The blue ones are the same for every model you add.

How to run the chat template gate in GitHub Actions (template-gate.yml)

The workflow runs the gate on every pull request. It installs the same pins, runs npm ci from Part 9's package-lock.json, and sets shell: bash so a pipe cannot hide the exit code. HF_HUB_OFFLINE is set because the gate needs nothing from the Hub. Every tokenizer it audits is a folder in the repository, so a Hub outage or a rate-limit error such as HTTP 429 cannot turn the build red, and nobody can change what the gate audits without changing the repository. To run it:

If chat-templates-part1/ is your repository's root, put the file at chat-templates-part1/.github/workflows/template-gate.yml. If the project sits in a subfolder of a larger repository, put the file in that repository's root .github/workflows/ and add working-directory: chat-templates-part1 to the defaults.run block.
Commit the code, qwen3-tokenizer/, smollm3-tokenizer/, gemma3-tokenizer/, goldens/, package.json and package-lock.json. Add gate.log, node_modules/ and .venv/ to .gitignore.
Open a pull request. The checks list shows template-gate / gate, and it should pass.
To watch it fail, push a branch with python studio_break.py backdoor applied and committed. The same check should go red, with the findings in its log.

yaml

# .github/workflows/template-gate.ymlname: template-gateon:  pull_request:  push:    branches: [main]jobs:  gate:    runs-on: ubuntu-latest    env:      HF_HUB_OFFLINE: "1"      TRANSFORMERS_NO_ADVISORY_WARNINGS: "1"    defaults:      run:        shell: bash  # bash -eo pipefail: a failing gate fails the step    steps:      - uses: actions/checkout@v7      - uses: actions/setup-python@v7        with:          python-version: "3.12"      - uses: actions/setup-node@v7        with:          node-version: "26"      - name: Install the pinned Python stack        run: |          pip install transformers==5.22.1 tokenizers==0.23.3 \            huggingface-hub==2.4.0 jinja2==3.1.6      - name: Install @huggingface/jinja from package-lock.json        run: npm ci      - name: Template gate        run: python studio_gate.py | tee gate.log

The three actions are pinned to their current major version, v7 (checkout, setup-python, setup-node). The gate does not depend on the action versions, so an older major tag works if your organisation has not allowed v7 yet. Two things about this file were not run here: GitHub Actions itself, and Node 26. The gate's code paths, the exit codes and the pipe behaviour are the ones you ran in steps 7 to 9. Three points match the GitHub documentation. Setting shell: bash adds pipefail. The runner marks a step as failed from its exit code. And actions/cache, if you add it later, saves a new cache only when the job succeeds (dependency caching reference), so a red gate never saves one.

Complete code for the chat template toolkit: studio.py, config and scripts

The full templatestudio/studio.py:

python

# templatestudio/studio.py"""One audit over a tokenizer's chat templates, and one exit code for CI."""from __future__ import annotationsimport textwrapfrom collections.abc import Collection, Mappingfrom dataclasses import dataclass, fieldfrom transformers import PreTrainedTokenizerBasefrom templatestudio.golden import (    CHAR_WINDOW,    GoldenCase,    GoldenFailure,    check_goldens,    load_goldens,)from templatestudio.parity import (    EngineError,    EngineRenderer,    chat_variables,    compare_engines,    lint_portability,)from templatestudio.security import (    assert_pinned,    scan_template,    template_fingerprint,    template_sources,)from templatestudio.render import load_tokenizerfrom templatestudio.serving import TokenRenderer, TransformersTokens@dataclass(frozen=True)class Finding:    check: str  # provenance, security, portability, golden, parity or reviewed    where: str  # the template or golden case the finding is about    detail: str@dataclassclass StudioReport:    findings: list[Finding] = field(default_factory=list)    @property    def failed(self) -> list[Finding]:        """Everything except findings someone reviewed for this exact template."""        return [f for f in self.findings if f.check != "reviewed"]    @property    def ok(self) -> bool:        return not self.faileddef template_findings(    name: str, source: str, reviewed: Mapping[str, Collection[str]]) -> list[Finding]:    """Lint and scan one template. reviewed maps a template fingerprint to    the scan findings someone read and accepted for exactly that source."""    accepted = reviewed.get(template_fingerprint(source), ())    findings = [Finding("portability", name, f) for f in lint_portability(source)]    for finding in scan_template(source):        check = "reviewed" if finding in accepted else "security"        findings.append(Finding(check, name, finding))    return findingsdef provenance_findings(    sources: Mapping[str, str], pins: Mapping[str, str]) -> list[Finding]:    """One finding per template that is unpinned or no longer matches its pin."""    findings = []    for name, source in sources.items():        if name not in pins:            findings.append(Finding("provenance", name, "no pin - review it first"))            continue        try:            assert_pinned(source, pins[name])        except ValueError as error:            findings.append(Finding("provenance", name, str(error)))    return findingsdef near(failure: GoldenFailure) -> str:    """What the golden expected and what came out, at the first difference."""    if failure.unit == "token":        return f"expected {failure.expected}, got {failure.actual}"    at = min(failure.position, CHAR_WINDOW)  # the difference, inside the window    expected, actual = failure.expected[at:at + 14], failure.actual[at:at + 14]    return f"expected {expected!r}, got {actual!r}"def golden_findings(    tokenizer: PreTrainedTokenizerBase,    goldens: list[GoldenCase],    engine: TokenRenderer | None = None,) -> list[Finding]:    """Part 11's golden check, one finding per half that left its golden."""    return [        Finding("golden", f.name, f"{f.unit} {f.position}: {near(f)}")        for f in check_goldens(tokenizer, goldens, engine=engine)    ]def parity_findings(    tokenizer: PreTrainedTokenizerBase,    goldens: list[GoldenCase],    renderers: list[EngineRenderer],) -> list[Finding]:    """Render every golden conversation on every engine and diff each engine    against the first one, with the template apply_chat_template would pick."""    findings = []    for case in goldens:        source = tokenizer.get_chat_template(tools=case.kwargs.get("tools"))        variables = chat_variables(tokenizer, case.conversation, **case.kwargs)        try:            reports = compare_engines(source, variables, renderers)        except EngineError as error:  # the reference engine refused            findings.append(Finding("parity", case.name, f"reference: {error}"))            continue        for report in reports:            if not report.identical:                where = report.error or f"char {report.first_divergence}"                findings.append(Finding("parity", case.name, f"{report.other}: {where}"))    return findingsdef audit_template(    tokenizer: PreTrainedTokenizerBase,    *,    pins: Mapping[str, str],    reviewed: Mapping[str, Collection[str]],    goldens: list[GoldenCase],    engine: TokenRenderer | None = None,    renderers: list[EngineRenderer] | None = None,) -> StudioReport:    """Every check that inspects a template, over every template it ships."""    sources = template_sources(tokenizer)    report = StudioReport(provenance_findings(sources, pins))    for name, source in sources.items():        report.findings += template_findings(name, source, reviewed)    report.findings += golden_findings(tokenizer, goldens, engine)    if renderers:        report.findings += parity_findings(tokenizer, goldens, renderers)    return reportALWAYS_BLOCK = frozenset({"provenance", "security", "golden"})def ci_gate(report: StudioReport, *, warn: Collection[str] = ()) -> int:    """Print every finding and return 1 if any finding outside warn failed.    Provenance, security and golden findings can never be made warnings."""    if not ALWAYS_BLOCK.isdisjoint(warn):        raise ValueError(f"these checks always block: {sorted(ALWAYS_BLOCK & set(warn))}")    blocking = [f for f in report.failed if f.check not in warn]    print("template audit:", "FAIL" if blocking else "PASS")    warnings = [f for f in report.failed if f.check in warn]    reviewed = [f for f in report.findings if f.check == "reviewed"]    for label, group in (("", blocking), ("warning: ", warnings), ("", reviewed)):        for f in group:            line = f"[{label}{f.check}] {f.where}: {f.detail}"            print(textwrap.fill(line, 86, initial_indent="  ", subsequent_indent=" " * 6))    return 1 if blocking else 0@dataclass(frozen=True)class ModelGate:    """Your application's side of the seam for one model: the tokenizer    folder it serves, the goldens recorded for it, and what was reviewed."""    tokenizer: str    goldens: str    pins: Mapping[str, str]    reviewed: Mapping[str, Collection[str]] = field(default_factory=dict)def audit_model(    gate: ModelGate, *, renderers: list[EngineRenderer] | None = None) -> StudioReport:    """Load what the service loads and audit it, token IDs included."""    tokenizer = load_tokenizer(gate.tokenizer)    return audit_template(        tokenizer,        pins=gate.pins,        reviewed=gate.reviewed,        goldens=load_goldens(gate.goldens),        engine=TransformersTokens(tokenizer),        renderers=renderers,    )

Your application's files:

python

# studio_models.py"""Your side of the seam: one entry per model you serve."""from security_pins import (    GEMMA3_PINS,    QWEN3_PINS,    QWEN3_REVIEWED,    SMOLLM3_PINS,    SMOLLM3_REVIEWED,)from templatestudio.studio import ModelGateMODELS = {    "qwen3": ModelGate(        tokenizer="qwen3-tokenizer",        goldens="goldens/qwen3.json",        pins=QWEN3_PINS,        reviewed={QWEN3_PINS["default"]: QWEN3_REVIEWED},    ),    "smollm3": ModelGate(        tokenizer="smollm3-tokenizer",        goldens="goldens/smollm3.json",        pins=SMOLLM3_PINS,        reviewed={SMOLLM3_PINS["default"]: SMOLLM3_REVIEWED},    ),    "gemma3": ModelGate(        tokenizer="gemma3-tokenizer",        goldens="goldens/gemma3.json",        pins=GEMMA3_PINS,    ),}# Every model here is served by Python Jinja2. Findings about the JS port# are printed as warnings; they do not fail the build.WARN = {"portability", "parity"}

python

# studio_gate.py"""CI entry point: audit every model you serve, exit 1 if any fails."""import sysfrom studio_models import MODELS, WARNfrom templatestudio.parity import CPythonJinja, NodeJinjafrom templatestudio.studio import audit_model, ci_gatedef main(names: list[str]) -> int:    status = 0    for name in names or list(MODELS):        print(f"== {name}")        report = audit_model(MODELS[name], renderers=[CPythonJinja(), NodeJinja()])        status = max(status, ci_gate(report, warn=WARN))    return statusif __name__ == "__main__":    sys.exit(main(sys.argv[1:]))

The new end of security_pins.py, after Part 12's QWEN3_PINS and QWEN3_REVIEWED:

python

SMOLLM3_PINS = {    "default": "3e0e275a527f249670056e0078e474eadbdd007c134b4fdfae5a7b3089702814",}GEMMA3_PINS = {    "default": "af95fbef33b76a50e5f463ff9766f85ce84f5849d915c4bd6c1619d852ac3231",}# Read in smollm3-tokenizer/chat_template.jinja. The scan reports kinds, not# places, so this covers every place each kind comes from. Content branches:# lines 18 and 20 pick the reasoning mode from /no_think or /think in the# system text; line 26 replaces the whole header with the system text when it# contains /system_override (not on the model card; only the system message# can trigger it). Long literals: lines 40 and 42, the default system prompts.SMOLLM3_REVIEWED = [    "long literal string injected into output",    "branch on message content - possible trigger condition",]

The scripts, in step order:

python

# studio_report.py"""What makes a report pass: no finding outside the reviewed ones."""from templatestudio.studio import Finding, StudioReportreport = StudioReport()print("empty report ok:", report.ok)report.findings.append(Finding("reviewed", "default", "long literal string"))print("reviewed finding only, ok:", report.ok)report.findings.append(Finding("golden", "plain", "token 9"))print("plus a golden finding, ok:", report.ok, "| failed:", len(report.failed))

python

# studio_lint.py"""Lint and scan the shipped template, then the same template plus one space."""from golden_cases import SHIPPEDfrom security_pins import QWEN3_PINS, QWEN3_REVIEWEDfrom templatestudio.render import load_tokenizerfrom templatestudio.studio import template_findingssource = load_tokenizer(SHIPPED).chat_templatereviewed = {QWEN3_PINS["default"]: QWEN3_REVIEWED}for label, template in (("shipped", source), ("plus one space", source + " ")):    print(f"== {label}")    for finding in template_findings("default", template, reviewed):        print(f"  {finding.check}: {finding.detail}")

python

# studio_provenance.py"""Check the shipped templates against their pins, then two changes."""from golden_cases import SHIPPEDfrom security_pins import QWEN3_PINSfrom templatestudio.render import load_tokenizerfrom templatestudio.security import template_sourcesfrom templatestudio.studio import provenance_findingssources = template_sources(load_tokenizer(SHIPPED))print("shipped:", provenance_findings(sources, QWEN3_PINS))changed = {"default": sources["default"] + " ", "tool_use": sources["default"]}for finding in provenance_findings(changed, QWEN3_PINS):    print(f"{finding.where}: {finding.detail}")

python

# studio_golden.py"""Part 11's goldens as findings: the shipped template, then one changed line."""from golden_cases import SHIPPEDfrom templatestudio.golden import load_goldensfrom templatestudio.render import load_tokenizerfrom templatestudio.serving import TransformersTokensfrom templatestudio.studio import golden_findingsfrom whitespace_commit import HUNKS, apply_hunkstokenizer = load_tokenizer(SHIPPED)goldens = load_goldens("goldens/qwen3.json")engine = TransformersTokens(tokenizer)print("shipped:", golden_findings(tokenizer, goldens, engine))tokenizer.chat_template = apply_hunks(tokenizer.chat_template, HUNKS[2:])findings = golden_findings(tokenizer, goldens, engine)print(f"undashed elif: {len(findings)} findings, the first two:")for finding in findings[:2]:    print(f"  {finding.where}: {finding.detail}")

python

# studio_parity.py"""Render Part 9's convert call on both engines, arguments as a string and a dict."""import jsonfrom amounts import ARGUMENTS, convert, convert_loopfrom golden_cases import SHIPPEDfrom templatestudio.golden import GoldenCasefrom templatestudio.parity import CPythonJinja, NodeJinjafrom templatestudio.render import load_tokenizerfrom templatestudio.studio import parity_findingsfrom templatestudio.tools import tool_schematokenizer = load_tokenizer(SHIPPED)tools = [tool_schema(convert)]cases = [    GoldenCase("string_args", convert_loop(ARGUMENTS), {"tools": tools}),    GoldenCase("dict_args", convert_loop(json.loads(ARGUMENTS)), {"tools": tools}),]findings = parity_findings(tokenizer, cases, [CPythonJinja(), NodeJinja()])print(f"{len(findings)} parity finding(s)")for finding in findings:    print(f"  {finding.where}: {finding.detail}")

python

# studio_audit.py"""Every template check on the shipped Qwen3 tokenizer, then the gate."""from golden_cases import SHIPPEDfrom security_pins import QWEN3_PINS, QWEN3_REVIEWEDfrom templatestudio.golden import load_goldensfrom templatestudio.parity import CPythonJinja, NodeJinjafrom templatestudio.render import load_tokenizerfrom templatestudio.serving import TransformersTokensfrom templatestudio.studio import audit_template, ci_gatetokenizer = load_tokenizer(SHIPPED)report = audit_template(    tokenizer,    pins=QWEN3_PINS,    reviewed={QWEN3_PINS["default"]: QWEN3_REVIEWED},    goldens=load_goldens("goldens/qwen3.json"),    engine=TransformersTokens(tokenizer),    renderers=[CPythonJinja(), NodeJinja()],)print("exit code:", ci_gate(report))

python

# studio_break.py"""Make one of three edits a pull request could make to the shippedtemplate file. Run golden_ship.py afterwards to put the original back."""import sysfrom pathlib import Pathfrom backdoors import REFUND_TRIGGER, insert_backdoorfrom golden_cases import SHIPPEDfrom whitespace_commit import HUNKS, apply_hunksTIDY = "{%- set messages = messages | reject('none') | list %}\n"BREAKS = {    "whitespace": lambda source: apply_hunks(source, HUNKS),    "reject": lambda source: TIDY + source,    "backdoor": lambda source: insert_backdoor(source, REFUND_TRIGGER),}name = sys.argv[1]path = Path(SHIPPED) / "chat_template.jinja"edited = BREAKS[name](path.read_text(encoding="utf-8"))path.write_text(edited, encoding="utf-8", newline="\n")print(f"{name}: edited {path.as_posix()}")

python

# studio_fstring.py"""Run the goldens through Part 1's f-string wrapper, then through Transformers."""from naive import build_prompt_naivefrom studio_models import MODELSfrom templatestudio.golden import load_goldensfrom templatestudio.render import load_tokenizerfrom templatestudio.serving import TransformersTokensfrom templatestudio.studio import audit_template, ci_gateclass FStringWrapper:    """Part 1's naive path: build the prompt with an f-string, then tokenize it."""    name = "f-string wrapper"    def __init__(self, tokenizer):        self.tokenizer = tokenizer    def ids(self, conversation, tools=None, **template_kwargs):        text = build_prompt_naive(conversation)        return self.tokenizer(text, add_special_tokens=False)["input_ids"]gate = MODELS["qwen3"]tokenizer = load_tokenizer(gate.tokenizer)# The wrapper never supported tools, so check it on the plain-chat goldens.goldens = [case for case in load_goldens(gate.goldens) if "tools" not in case.kwargs]for engine in (FStringWrapper(tokenizer), TransformersTokens(tokenizer)):    print(f"== goldens through {engine.name}")    report = audit_template(        tokenizer, pins=gate.pins, reviewed=gate.reviewed, goldens=goldens, engine=engine    )    ci_gate(report)  # a demo, not a CI entry point: the exit code is ignored

python

# studio_record.py"""Save a model's tokenizer into the folder your service loads, record itsgoldens, and print the pins to paste into security_pins.py."""import sysfrom golden_cases import CASESfrom templatestudio.golden import record_golden_ids, record_goldens, save_goldensfrom templatestudio.render import load_tokenizerfrom templatestudio.security import template_fingerprint, template_sourcesfrom templatestudio.serving import TransformersTokensmodel_id, folder, goldens_path, *names = sys.argv[1:]load_tokenizer(model_id).save_pretrained(folder)tokenizer = load_tokenizer(folder)cases = record_goldens(tokenizer, [case for case in CASES if case.name in names])cases = record_golden_ids(TransformersTokens(tokenizer), cases)save_goldens(goldens_path, cases)print(f"{len(cases)} goldens -> {goldens_path}")for name, source in template_sources(tokenizer).items():    print(f'    "{name}": "{template_fingerprint(source)}",')

python

# studio_policy.py"""Try to turn a security finding into a warning."""from studio_models import MODELSfrom templatestudio.studio import audit_model, ci_gatereport = audit_model(MODELS["smollm3"])try:    ci_gate(report, warn={"parity", "security"})except ValueError as error:    print(f"ValueError: {error}")

The files this part adds or changes, inside the project from Parts 1 to 12:

text

chat-templates-part1/├── .github/workflows/template-gate.yml   new: runs studio_gate.py on every pull request├── templatestudio/studio.py              new: Finding, StudioReport, audit_template,│                                         ci_gate, ModelGate, audit_model├── studio_models.py                      new: MODELS and WARN, your side of the seam├── studio_gate.py                        new: the CI entry point├── security_pins.py                      changed: SMOLLM3_PINS, GEMMA3_PINS,│                                         SMOLLM3_REVIEWED├── goldens/smollm3.json                  new, written by studio_record.py├── goldens/gemma3.json                   new, written by studio_record.py├── smollm3-tokenizer/                    new, written by studio_record.py├── gemma3-tokenizer/                     new, written by studio_record.py├── gate.log                              written by the tee runs; do not commit└── studio_report.py, studio_lint.py, studio_provenance.py, studio_golden.py,    studio_parity.py, studio_audit.py, studio_break.py, studio_fstring.py,    studio_record.py, studio_policy.py

Next: bump to Transformers 5.22.2 under the gate and add non-template checks

Run the checks that test something other than the template as a second CI step. The gate is limited to the template. Add one script beside studio_gate.py that exits 1 when Part 4's system_handling does not return the answer you expect for a model (or role_handling for another role you send), when Part 8's check_stop_alignment finds the end-of-turn token missing from the generation config's eos_token_id, or when Part 10's assert_prefix_stable raises for a recorded session. Each is already a function in your project, but each needs a fixture the gate does not carry: an expected answer, a generation config, a session.
Bump transformers to 5.22.2 under the gate. Change the pin in the workflow and in your environment, then run python studio_gate.py. The sandbox change in PR #50431 changes how template data is passed in. If a golden moves, the gate names the case and the token. If nothing moves, you have measured that the upgrade changes nothing you test. Do the same before every Transformers, Node or @huggingface/jinja upgrade.
Plan for templates written as code. A few 2026 models ship no Jinja template. vLLM serves Kimi K3 through KimiK3Renderer, which calls the model's own Python encoder. PrimeIntellect's renderers library builds token IDs in Python with render_ids, and it falls back to apply_chat_template for any model it does not cover. For those models the lint, the scan and Jinja parity have nothing to read. A pin on the renderer's version and token-level goldens still work, with some wiring. Wrap the renderer in a class with name and ids(), record goldens with no expected string, and call check_goldens directly. audit_template itself stops on such a tokenizer, because template_sources finds no template to pin.

Chapter 13 of The Chat Templates Handbook, "Building a Template Toolkit", covers this part's ground with the full toolkit around it: the integration contract as a boundary for the whole fleet, the production considerations for severity and for growing the golden set from incidents, and post-render instrumentation such as assistant-token masks. The companion repository's v1.2/ch13-capstone tag has the book's version of studio.py. It differs from this one. It audits a template string passed beside the tokenizer, and it keeps one finding list per check.

That ends the series. Part 1 started with one rendered prompt that nobody had looked at. Part 13 ends with every template you serve fingerprinted, golden-tested on its token IDs, scanned, checked on a second engine, and gated in every pull request.

References

Kumar, R. (2026). The Chat Templates Handbook, chapter 13, "Building a Template Toolkit".
Template Studio companion repository, tag v1.2/ch13-capstone.
Transformers v5.22.1 (used here) and v5.22.2.
Transformers apply_chat_template reference: template selection, tools, return_assistant_tokens_mask.
Transformers PR #50431, "Harden our Jinja sandboxing", in 5.22.2.
Transformers: chat templates, writing a template: named templates and tool_use selection.
Jinja2 3.1.6 on PyPI and tokenizers 0.23.3 on PyPI.
huggingface_hub environment variables: HF_HUB_OFFLINE, version 2.4.0.
Hugging Face Hub rate limits.
Python 3.12 sys.exit and textwrap.fill.
GitHub Actions workflow syntax: defaults.run.shell and exit codes.
GitHub Actions: building and testing Python.
github/docs issue #23853: the default shell runs without pipefail.
GitHub Actions dependency caching reference.
actions/checkout v7.0.1, actions/setup-python v7.1.0, actions/setup-node v7.1.0, and actions/cache (it saves a cache only when the job succeeds, per the dependency caching reference above).
Node.js release schedule.
@huggingface/jinja 0.5.12 on npm.
Qwen/Qwen3-0.6B model card, SmolLM3-3B model card and unsloth/gemma-3-270m-it.
vLLM KimiK3Renderer.
PrimeIntellect renderers.

AI Engineering

How to Detect Chat Template Backdoors and Forged Tokens

Llms

Follow for more technical deep dives on AI/ML systems, production engineering, and building real-world applications:

How to Gate Chat Template Changes in CI with One Audit

What you'll build: one CI gate for every chat template you serve

Prerequisites for the chat template CI gate: the project from Parts 1 to 12

How to structure a chat template CI gate: one ModelGate record per model

Step 1: Define a report that every chat template check writes to

Step 2: Lint and scan a chat template with reviews bound to its fingerprint

Step 3: Pin and check every Hugging Face chat template by fingerprint

Step 4: Add golden-token tests for a chat template to the audit

Step 5: Find Jinja engine parity failures for every golden conversation

Step 6: Combine the checks into audit_template and a CI gate exit code

Step 7: Give CI one entry point: the model config and studio_gate.py

Step 8: Test that the CI gate fails on whitespace, filter and backdoor edits

Step 9: Fix a green CI step that hides a failing gate (tee and pipefail)

Step 10: Run Part 1's f-string prompt through the gate's golden-token test

Step 11: Add a second and third model to the CI gate with one config entry each

Step 12: Review SmolLM3's scan findings against its fingerprint

Step 13: Set a CI severity policy: which chat template checks may only warn

When it breaks: chat template CI gate errors and their fixes

Fix: OSError "We couldn't connect to 'https://huggingface.co'" when the gate runs offline

Fix: FileNotFoundError WinError 2 when the gate starts Node

Fix: AssertionError "hunk not found once" from studio_break.py

Error: Unknown ObjectValue filter: string in @huggingface/jinja

How the chat template CI gate fits together

How to run the chat template gate in GitHub Actions (template-gate.yml)

Complete code for the chat template toolkit: studio.py, config and scripts

Next: bump to Transformers 5.22.2 under the gate and add non-template checks

References

Books by Ranjan Kumar

Federated AI Agents

Right-Sized AI

Harness Engineering

The 7 GenAI Architectures

Building Real-World Agentic AI Systems

The ChatML Handbook

The Chat Templates Handbook

Comments

What you'll build: one CI gate for every chat template you serve

Prerequisites for the chat template CI gate: the project from Parts 1 to 12

How to structure a chat template CI gate: one ModelGate record per model

Step 1: Define a report that every chat template check writes to

Step 2: Lint and scan a chat template with reviews bound to its fingerprint

Step 3: Pin and check every Hugging Face chat template by fingerprint

Step 4: Add golden-token tests for a chat template to the audit

Step 5: Find Jinja engine parity failures for every golden conversation

Step 6: Combine the checks into audit_template and a CI gate exit code

Step 7: Give CI one entry point: the model config and studio_gate.py

Step 8: Test that the CI gate fails on whitespace, filter and backdoor edits

Step 9: Fix a green CI step that hides a failing gate (tee and pipefail)

Step 10: Run Part 1's f-string prompt through the gate's golden-token test

Step 11: Add a second and third model to the CI gate with one config entry each

Step 12: Review SmolLM3's scan findings against its fingerprint

Step 13: Set a CI severity policy: which chat template checks may only warn

When it breaks: chat template CI gate errors and their fixes

Fix: OSError "We couldn't connect to 'https://huggingface.co'" when the gate runs offline

Fix: FileNotFoundError WinError 2 when the gate starts Node

Fix: AssertionError "hunk not found once" from studio_break.py

Error: Unknown ObjectValue filter: string in @huggingface/jinja

How the chat template CI gate fits together

How to run the chat template gate in GitHub Actions (template-gate.yml)

Complete code for the chat template toolkit: studio.py, config and scripts

Next: bump to Transformers 5.22.2 under the gate and add non-template checks

References

Related Articles

Books by Ranjan Kumar

Federated AI Agents

Right-Sized AI

Harness Engineering

The 7 GenAI Architectures

Building Real-World Agentic AI Systems

The ChatML Handbook

The Chat Templates Handbook

Comments