← Back to Guides

Companion to the book Harness Engineering for Production AI Systems · this part maps to Book Ch 4

GuideFor: AI Engineers, ML Engineers, Platform Engineers, AI Security Engineers

LLM Input Validation: Judge the Record, Not the Clean Input

The input failure no detector is built for needs no attacker: your own words come back under a customer's label. Split the entry point into a sensor that records what it changed and a gate that never cleans.

#harness-engineering#llm-input-validation#prompt-injection#unicode-normalization#llm-security#production-ai-systems

In an incident posted on 12 September 2024 (UTC), the customer support platform Gorgias wrote that "under specific conditions, our AI Agent encountered a loop causing it to respond to its own outputs. This resulted in an undesirable behavior affecting some customer interactions" (Gorgias status page). Gorgias published no root cause and named no attacker. What I take from it is the shape: an agent read its own words as if somebody else had written them. It is also the failure that conventional LLM input validation, the cleaning and filtering at an agent's entry point, is not built to catch.

Here is that shape in the billing support agent this series and its book are built around. A customer replies to an email, and their mail client quotes the thread underneath, as mail clients have done for decades.

text
I still haven't seen the money. You said it was approved two weeks ago.--- Original Message ---On 12 August, Billing Support wrote:> Thanks for getting in touch. I have approved a refund of $940.00 against your> account and it should reach you within five working days.

Those quoted lines are the agent's own draft from 12 August, which a tier-1 employee read, trimmed and sent. It arrives in the ticket_body field, so it is tagged customer. That tag is true and useless. On the account itself, the only open invoice is $120, and every turn the agent now runs there includes a sentence, written in the system's own voice, saying $940 was approved. Asked where the money is, the agent has every reason to tell the customer $940 is on its way. Filing our words under customer looks like the cautious, lower-trust direction. What makes it dangerous is the voice: the sentence is phrased as the system speaking, and the model is not bound by the label.

Run the usual input defense over that ticket and it passes everything. My own April version of this argument on this blog promised a layer that "ensures the model only ever sees clean, controlled, safe input", and opened its pipeline with a regex for phrases like "ignore all previous instructions". Nothing here matches. An injection classifier is not built to flag it either: by its own model card, Prompt Guard 2 looks for explicit attempts to override instructions, and nothing here tries to override anything. This text is clean; it is just in the wrong place with the wrong label.

My claim is that an LLM entry point should be judged by the record it leaves, not by how clean the model's input looks. The failure that clean-input defenses are not built for needs no attacker: your own output re-enters under a customer's label, a pattern I call Laundered Provenance. And every lossy cleaning step destroys evidence that no later step in the pipeline can recover unless you kept the exact bytes, because the raw bytes are observable only inside what I call the Normalisation Window, the interval between receiving input and producing canonical text.

So "input validation" has to be split into two components: a sensor that never refuses and records, in code points, what every step changed, and a gate that refuses, decodes once and never cleans. Origin comes from the channel the text arrived on, the raw text stays in the system of record, and a classifier may sit at the entry point as a sensor but never as the thing that says no. The fix for the opening ticket still changes what the model reads. The record is what makes that change reviewable, and it is the only way to tell a fix from a deletion.

Part 2 called this the Flat Context: the model reads one flat sequence, so an origin label is something the harness enforces, not something the model respects. Its Context.add(origin, text) is where the label gets attached, and at an entry point keyed on fields, the field name is all it uses. Part 3 sorted components by what they can stop. This part applies both to the first component every request touches.

Why LLM input validation that only cleans misses the failure that needs no attacker

Ask what an input validation layer does and the answer usually mixes two kinds of verb. It strips control characters, normalises Unicode, collapses whitespace. It rejects oversize payloads, refuses unknown fields, enforces a maximum length.

Part 3's Role Test splits that list on one question: can this component make the request fail? Stripping and normalising cannot. They change what the model reads and pass it on, which makes them a sensor. Rejecting and refusing can, which makes them a gate. Libraries ship the two merged. LLM Guard's InvisibleText scanner, from a project archived on GitHub, returns (sanitized_prompt, is_valid, risk_score) from one call, so the same function removes characters and refuses (LLM Guard). Mastra's UnicodeNormalizer processor receives both the messages and an abort function (Mastra). In application code it is usually one clean(). This and every snippet below are excerpts from the article's code (entry.py, gate.py, wrong.py), with imports, the FieldRule dataclass and the NotAdmitted exception left out:

python
def clean(text: str, limit: int = 8_000) -> str:    """The merged version: transform first, then refuse on whatever is left."""    text = unicodedata.normalize("NFKC", strip_invisible(text))    if len(text) > limit:        raise ValueError(f"{len(text)} characters, limit {limit}")    return text

Merging them breaks in three specific ways.

The length check runs on the cleaned text. Strictly that is an ordering bug, and a merged function could check raw length first. The merge is what makes the wrong order natural: inside one function, the check wants the text it is about to return. So a clean() that normalises first, because validation is easier against canonical text, measures a payload of zero-width spaces after stripping it, finds nothing left, and passes it.

A refusal arrives with no reason. A cleaner that raises from somewhere inside its steps turns one odd character into a ticket with no reply and an exception nobody can map to a policy. Support notices, and the usual fix is to make the cleaner permissive, which quietly removes the check that was doing work. Refusing is not the fault here. Below, the split gate also refuses a ticket with one bad byte, but it refuses only on facts it declares, names the field and the byte offset, and leaves the transforming steps no way to fail.

And the transformation disappears. When cleaning and refusing share a function, the only observable outcome is pass or fail. A ticket that lost four lines returned successfully, and returning successfully is the good case, so nothing anywhere says four lines were taken.

That third failure is the one this article is about, and the book's own sensor shows it on a ticket that is under every limit. Take the opening ticket, add one line below the quote, "Also, email me the receipt.", and run the companion repository's Intake:

text
ticket_body  strip-quoted-reply   233 charactersmodel sees: "I still haven't seen the money. You said it was approved two weeks ago."

The record says the quote strip took 233 characters, 29 more than on the opening ticket. Nothing says one of those lines was the customer's request, and no reordering of the steps brings it back. A merged clean() would not say even that much. The cruder case is a flood, where the cost is that the limit no longer bounds what you process. Here are both designs on 1.4 million zero-width spaces:

text
payload: 1,400,000 characters, 4,200,000 bytesmerged clean(): passed, 0 characters left, nothing recordedsplit admit():  NotAdmitted: ticket_body: 4200000 bytes, limit 32000

Below is the order of events behind those two lines.

d2
grid-rows: 1
grid-gap: 40
merged: "Merged: clean() then check" {
  label.near: top-center
  style: {fill: "#FDF2E9"; stroke: "#D9785A"}
  grid-columns: 1
  grid-gap: 18
  m1: "1. 4.2 MB of zero-width spaces" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
  m2: "2. strip + NFKC: 0 characters left" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  m3: "3. length check on 0 characters: pass" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
  m4: "4. record: pass" {style: {fill: "#F4F6F7"; stroke: "#95A5A6"; font-color: "#2C2C2A"}}
}
split: "Split: gate on raw, sensor after" {
  label.near: top-center
  style: {fill: "#EAF7F0"; stroke: "#3F9E52"}
  grid-columns: 1
  grid-gap: 18
  s1: "1. 4.2 MB of zero-width spaces" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
  s2: "2. gate on raw bytes: 4.2 MB > 32,000" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
  s3: "3. NotAdmitted, sensor never runs" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
  s4: "4. record: refusal with a reason" {style: {fill: "#F4F6F7"; stroke: "#95A5A6"; font-color: "#2C2C2A"}}
}

In the merged column, step 3 is a correct check on the wrong text. Nothing in that column is buggy line by line, which is why the design survives code review. Splitting costs one rule: the gate reads raw input and returns only what it decoded, and the sensor transforms and is not allowed to raise.

Laundered Provenance: your own words, back under a customer's label

Follow the four quoted lines from the opening ticket. Our system wrote them. An employee edited and sent them. A mail client quoted them, and the customer posted them back as a ticket body. At every hop the text was the same. Only its label changed, from "the system said this" to "a customer said this", and nothing recorded the change.

That is the definition. Laundered Provenance is text that re-enters your system through a channel that gives it a new origin, weaker or stronger than the one it had, with no record of the change. Its commonest form is your own output coming back as somebody's input, and it needs no attacker.

d2
direction: down
classes: {
  hop: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  label: {style: {fill: "#F4F6F7"; stroke: "#95A5A6"; font-color: "#2C2C2A"}}
}
grid-rows: 4
grid-columns: 2
grid-gap: 22
h1: "1. Agent drafts the reply" {class: hop}
l1: "origin: system" {class: label}
h2: "2. Employee trims and sends it" {class: hop}
l2: "origin: system (outbound mail)" {class: label}
h3: "3. Customer's client quotes it" {class: hop}
l3: "origin: no longer recorded" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
h4: "4. Arrives in ticket_body" {class: hop}
l4: "origin: customer, from the field name" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
h1 -> h2
h2 -> h3
h3 -> h4

The right-hand column is the whole problem. Hop 3 is where the record is lost, and hop 4 assigns a new origin from the only fact a field-keyed entry point uses, which is the field the text arrived in. That is Part 2's Context.add doing exactly what it was told. It also skews Part 2's untrusted fraction, which, before any strip runs, counts 204 characters, most of them our own reply, as customer text.

The direction decides which failure you get. Downgraded, as here, your words come back labelled customer. A model that reads "Billing Support wrote: I have approved a refund of $940" has no way to tell a customer's claim about history from history. Upgraded, customer text lands in a field your code tags operator or tool-result, and everything that trusts those origins now trusts a stranger. The book calls the upgrade the dangerous direction, and for an attacker it is. The downgrade is the one nothing alerts on.

Every well-documented case I could find runs in the upgrade direction, with an attacker. In EchoLeak (CVE-2025-32711), an external email entered Microsoft 365 Copilot's context with internal authority, and the attack chained bypasses that included "evading Microsofts XPIA (Cross Prompt Injection Attempt) classifier" (Reddy et al., 2025). In General Analysis's Supabase MCP demonstration, a support ticket was read by an assistant holding service_role privileges: "The weak link: the IDE assistant ingests untrusted customer text and holds service_role privileges" (General Analysis). That one was dummy data, not a customer incident, and it is the closest public match to a billing agent I know of. Part 11 follows an injected ticket into a refund call.

Sharpest of all is a paper from October 2026 that names a special case of this pattern. Arshan Dabirsiaghi's compaction laundering forges a user-role line that a coding agent's context compactor then promotes into memory: "Across five flattening harnesses (pi, opencode, Cline, Aider, OpenHands) the forged approval launders into the agent's memory 87-100% of the time" and "across four role-typed harnesses (Codex, Gemini CLI, Qwen Code, Continue) the identical attack succeeds 0% of the time" (Dabirsiaghi, 2026). His conclusion is the one this section depends on: "Because the provenance is destroyed deterministically in harness code, no prompt-level defense can restore it". Compaction laundering is the attacker-driven, upgrade-direction member of the class. At the other end sits the billing ticket: no attacker, downgrade direction, and nothing that any detector is looking for.

I am not going to claim the no-attacker case is the most consequential, because the public record does not support that: Gorgias is the only no-attacker incident I found, and its write-up is two sentences. I will defend a narrower claim. I expect it is the commonest form, because any mail client that quotes the thread produces it every time a customer replies to you, and it is under-reported because nothing alerts on it. An injection that gets caught leaves a log line. A laundered approval leaves a reasonable-looking reply. Some helpdesks already cut quoted history at a "reply above this line" delimiter, so the laundering happens where your intake bypasses that cut or the delimiter fails, and those cutters also decide by format and record no origin.

Assign origin from the channel: In-Reply-To beats a quote regex

As a general defense, this one is not clever: the origin tag has to come from the channel the text arrived on, and the channel has to be narrow enough for the tag to mean something. A ticket_body field fed by one inbound path is a narrow channel. A notes field that three subsystems write to is not, and no downstream logic recovers what was lost when that field was designed.

For quoted email, the book's Chapter 4 uses a regex step, strip-quoted-reply, that deletes any block shaped like a quote. It works on the opening ticket, removing 204 characters. It also decides origin from format, and that has two costs. It deletes quotes the customer meant to include, whoever wrote them. The channel version below narrows that to replies to us, though it still drops every > line in such a reply, including a bank email the customer quoted alongside ours. And its pattern for "On ... wrote:" runs with re.DOTALL to the end of the text, so anything the customer typed below the quote goes too.

Email already carries the channel fact. When a customer replies, their client sets the RFC 5322 In-Reply-To header to the Message-ID of the message they are answering. You sent that message, so it is in your outbound store with the exact text you sent. Nylas's agent cookbook uses the same idea for loops: its first guard skips "Another agent's grant_id, or the agent's own message", deciding "own" from sender identity rather than from what the text looks like (Nylas).

python
@dataclass(frozen=True)class Inbound:    body: str    in_reply_to: str | None            # RFC 5322 header, set by the customer's client    sender: str                        # the address the ticket came from    references: tuple[str, ...] = ()   # RFC 5322 References: the rest of the threaddef _unquote(line: str) -> str:    return line.lstrip().lstrip("> ")              # handles nested >> quoting toodef _canon(text: str) -> str:    return " ".join(unicodedata.normalize("NFKC", text).split())def _is_attribution(line: str) -> bool:    s = line.strip().lower()    return s.startswith(("---", "from:", "sent:", "date:")) or (        s.startswith("on ") and s.endswith("wrote:")    )def split_own_quote(    msg: Inbound, sent: Mapping[tuple[str, str], str]) -> tuple[str, str | None]:    """Remove a quoted block only when the channel says it is a reply to us.    `sent` is keyed by (recipient, Message-ID), so a reply can only match mail we sent    to this sender. Returns the customer's own text and a verdict for the record:    None when the message replies to nothing we sent this sender,    "reply-no-quote-found" when it does but no `>` block is present,    "own-quote" when the quote contains what we sent, and    "quote-mismatch" when it replies to us but our words are not in the quote.    """    ids = [msg.in_reply_to, *msg.references]    ours = next((sent[(msg.sender, i)] for i in ids if (msg.sender, i) in sent), None)    if ours is None:        return msg.body, None                       # not a reply to us: all customer text    lines = msg.body.splitlines()    quoted = [i for i, line in enumerate(lines) if line.lstrip().startswith(">")]    if not quoted:        return msg.body, "reply-no-quote-found"     # say so; never look like None    head = quoted[0]    while head > 0 and _is_attribution(lines[head - 1]) and head > quoted[0] - 2:        head -= 1                                   # only lines shaped like a header    quote = " ".join(_unquote(lines[i]) for i in quoted)    verdict = "own-quote" if _canon(ours) in _canon(quote) else "quote-mismatch"    drop = set(range(head, quoted[0])) | set(quoted)    kept = "\n".join(line for i, line in enumerate(lines) if i not in drop)    return kept.strip(), verdict

The verdict goes into the sensor's record beside a Change for the split, measured in code points like every other step. Finding the quoted lines still uses their > prefix, which is format. What changed is who decides the origin. In-Reply-To says which of your messages this answers, and your outbound store says what that message contained. Nine runs, with the 12 August reply stored under the customer's address and its Message-ID:

text
reply to us    verdict=own-quote            kept=71 removed=204no In-Reply-To verdict=None                 kept=275 removed=0edited quote   verdict=quote-mismatch       kept=71text below     verdict=own-quote            kept='Also, email me the receipt.'other account  verdict=None                 kept=275no > quote     verdict=reply-no-quote-found kept=129signature too  verdict=own-quote            kept=71line above     verdict=own-quote            kept='Please help'References     verdict=own-quote            kept=71

Line one matches the book's number, 204 characters, now with a reason attached: this was our own message, by Message-ID. For a message that replies to nothing we sent, the second line is the correct answer, and the fifth line gives the same answer when another account points In-Reply-To at our Message-ID, because the store is keyed by recipient as well. A forged "Billing Support wrote:" block with no matching header stays as customer text, because that is exactly what it is, and the refund gate from Part 3 that checks the ledger is what keeps it from becoming a payment. The fourth line keeps the request the regex would have deleted. The sixth is the case a format check misses silently: an Outlook-style reply quotes our message under a From: header with no > prefix, so nothing is removed, and the record says reply-no-quote-found instead of looking like a message that was never a reply. The sentence still reaches the model in that case, so the defense there is the record plus the reply check downstream, not the entry point. A containment search for our sent text in the whole body would close it, at the cost of one more scan.

Format matching cannot produce the third line. In that run, the customer replied to our message and changed $940.00 to $9,400.00 inside the quote. We still remove the quote, and the record now says the customer's copy of our words differs from what we sent, which may be an upgrade attempt, caught by comparing against a store the customer cannot write. It may also be a mail client that converted HTML to text or curled the quotation marks, so quote-mismatch is evidence for a person to look at, never a refusal, and its rate on real traffic is something I have not measured.

The last three lines are the matcher bending where real mail needs it. A quote that also carries our signature still matches, because the check is containment after NFKC and whitespace folding, which also absorbs re-wrapped lines and nested >> quoting. A customer's own line directly above the quote survives, because only lines shaped like a mail header count as attribution. And a client that sets only References, which carries the rest of the thread, still finds the message.

d2
grid-rows: 4
grid-columns: 2
grid-gap: 24
q1: "1. Reply headers name mail sent to this sender?" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
a1: "No: keep everything, origin customer" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
q2: "2. A > quoted block found?" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
a2: "No: keep it, record reply-no-quote-found" {style: {fill: "#F4F6F7"; stroke: "#95A5A6"; font-color: "#2C2C2A"}}
q3: "3. Quote contains what we sent?" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
a3: "Yes: drop it, record own-quote" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
q4: "4. Otherwise" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
a4: "Drop it, record quote-mismatch" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
q1 -> q2
q2 -> q3
q3 -> q4

Every outcome on the right is a record entry rather than a refusal, because this is the sensor. What an own-quote or a quote-mismatch means for a refund is decided later, where the proposal exists.

Two limits. In-Reply-To is set by the sender's client, so it is a claim, not a proof: a customer can point it at any Message-ID they received. That is fine here for two reasons. The header only selects which of our messages to compare against, and the comparison is against our store. And the store is keyed by recipient and Message-ID, so a customer who points the header at another account's mail gets no match and nothing from that account is read, provided the sender address is authenticated. Key on it only after SPF, DKIM and DMARC pass, or on the helpdesk's customer ID. Keep that scoping if you also retrieve from the store, or the lookup becomes a way to pull one customer's mail into another's context. If tickets reach you through a helpdesk API rather than as raw mail, check whether it passes In-Reply-To through. If it drops the header, that is the field to ask for. And a sentence the system wrote is still not a fact about the ledger. own-quote means "we said this on 12 August", which is not evidence that a refund of $940 was owed. The ledger answers that.

Why drop a verified quote instead of re-attaching it as origin: system? Because the customer's copy is still the customer's copy. The authoritative text already sits in the outbound store under its Message-ID, with the date it was sent. If the reply needs it, retrieve it from there.

The Normalisation Window: evidence that exists only before canonical text

Every normalisation step is lossy by design: you discard information so that what remains is comparable, and the question is what else you discarded.

Standard practice says canonicalise first. CWE-180 is titled "Incorrect Behavior Order: Validate Before Canonicalize" and describes the weakness as "The product validates input before it is canonicalized" (MITRE). OpenSSF's Python guide carries the same rule as pyscg-0044 (OpenSSF). That rule is right, and this article does not argue with it. A check for the word refund should run on text where NFKC (UAX #15) has already turned fullwidth refund into refund, or it will miss.

What the rule does not cover is a second kind of check. Content validation asks whether the text says something forbidden, and it belongs after canonicalising. Anomaly evidence asks whether the sender did something unusual to the bytes: fullwidth letters in an English ticket, invisible tag characters, a bidirectional override, a replacement character from a failed decode. Inside the pipeline, that evidence exists only before canonicalising, because canonicalising is the act of removing it. Unicode's own security report describes the hazard from the other side: a check passes, and a later process invisibly deletes a character so that the forbidden string forms after the check ran (UTR #36, section 3.5). OpenSSF's guide puts it in one line, under CWE-182: "stripping of characters can lead to a harmless string such as <script生> turn into <script>".

So: the Normalisation Window is the interval between receiving raw input and producing canonical text. Inside it, the original bytes still exist and anything unusual about them is observable. After it closes, the input is comparable and the evidence is gone. It closes exactly once per input, per field, which in an agent loop means again for every tool result and retrieved document.

UTR #36 already warns against deleting after a check, so the hazard is not new. What the Window adds is an operating rule: read anomaly evidence once, on the raw text, before the first transform, and log it beside each step's changes, because no later step in the pipeline can read it.

d2
direction: down
grid-rows: 2
grid-columns: 4
grid-gap: 20
w1: "raw bytes" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
w2: "decoded text" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
w3: "each step runs" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
w4: "canonical text" {style: {fill: "#F4F6F7"; stroke: "#95A5A6"; font-color: "#2C2C2A"}}
c1: "gate: size, fields,\nstrict decode" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
c2: "sensor: evidence\n(tags, bidi, fullwidth)" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
c3: "sensor: what each\nstep changed" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
c4: "content checks\n(CWE-180's place)" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
w1 -> w2
w2 -> w3
w3 -> w4

Read the bottom row as "what can be seen here": the first three columns are inside the window, and only the fourth is where CWE-180 puts validation. Moving a check that reads bytes or characters from the bottom-left three into the fourth column makes it blind, inside the pipeline, to what it was looking for.

Order inside the window matters for the same reason. A zero-width joiner between a letter and a combining accent stops NFKC from composing them. Take cafe followed by U+200D and a combining acute accent:

text
nfkc first: 5 normal=Falsestrip first: 4 normal=True

Normalise first and the joiner blocks composition (a zero-width joiner has combining class 0, which blocks under definition D115 in section 3.11 of the Unicode core spec), then the strip removes it, leaving an e and a loose accent in text your pipeline now calls canonical. Strip first and you get a four-character café in normal form. Same steps, same input, and a substring check for café passes on one ordering only. Write a test that pins each ordering you rely on, because nothing else will.

Count what each step changed, not how much shorter the text got

The book's sensor records each step's losses as a Removal(field, step, characters), where characters is the length before minus the length after. That is the right instinct and the wrong unit. I ran the companion repository's Intake (commit d52b6ea) over four inputs, using the ticket_body steps:

InputBook's recorded charactersWhat actually happened
please refund me (fullwidth)nothing recordedNFKC replaced 6 code points
wait… what-2NFKC turned 1 code point into 3
refund plus 11 Unicode tag charactersnothing recordednothing removed; the payload is still there
a woman technologist emoji, 👩‍💻1, by strip-controlthe ZWJ that joins the emoji was deleted

In the first row, NFKC does the folding it exists to do, and a length delta cannot see it, because fullwidth-to-ASCII is one character for one. Row two records a negative removal. Both are exactly the evidence the record exists to keep. Measuring changed code points in both directions fixes it:

python
@dataclass(frozen=True)class Change:    """What one step did to one field, in code points. Length cannot see a swap."""    field: str    step: str    removed: int    inserted: intdef measure(field: str, step: str, before: str, after: str) -> Change | None:    """Code points as a multiset: linear time, and a one-for-one swap still shows."""    b, a = Counter(before), Counter(after)    removed, inserted = sum((b - a).values()), sum((a - b).values())    return Change(field, step, removed, inserted) if removed or inserted else None

I tried difflib.SequenceMatcher first, which gives exact edit positions, with autojunk=False because its default heuristic treats frequent characters as junk and misses matches on repetitive text. On one 8,000-character field of fullwidth text it took 60.8 seconds in the run captured for this article. Counting code points as a multiset took under 2 ms on the same field. It cannot see a step that only moves characters around, and the only reordering in this pipeline is NFKC's canonical ordering of combining marks, which a multiset correctly ignores. Alongside the per-step changes, the sensor takes one reading of the raw text before any step runs, because that is the only moment it can:

python
def _kind(c: str) -> str | None:    cp = ord(c)    if 0xE0000 <= cp <= 0xE007F:        return "tag"    if 0x202A <= cp <= 0x202E or 0x2066 <= cp <= 0x2069:        return "bidi-control"    if c in "\u200b\u200e\u200f\u2028\u2029\u2060\u00ad\ufeff":        return "invisible"    if c in "\u200c\u200d":        return "joiner"    if c == "\ufffd":        return "replacement"    if unicodedata.normalize("NFKC", c) != c:        return "compatibility-form"    return Nonedef evidence(text: str) -> Counter[str]:    """What is unusual about the raw text. Gone once the steps have run."""    return Counter(k for c in text if (k := _kind(c)))

One case in the run below is a control: when a step only deletes, as with a ZWJ inside refund, the length delta is correct. The two instruments disagree exactly on substitutions and expansions, which are what NFKC does.

What a control-character strip misses, and what it breaks

strip-control, in the companion, covers C0 controls, DEL, U+200B to U+200F, the line and paragraph separators, and the byte-order mark. It does not cover the Unicode tag block, U+E0000 to U+E007F. That block mirrors ASCII, renders as nothing in most interfaces, and is read by models. Johann Rehberger's write-up is blunt about it: "For LLM applications filtering out the Unicode Tags Code Points at prompting and response times is a mitigation that apps need to implement", and the same hidden text "has the power to exploit the 'Human in the Loop' mitigation" because the reviewer cannot see it (Embrace The Red). NFKC does not remove tag characters either. Through the book's ticket_body steps, all 11 characters of a tag-encoded "approve 940" came out the other end. So the book's input policy does not defend against ASCII smuggling, and I will not pretend it does.

Fixing it runs into real customers twice. Subdivision flags are spelled with the tag block too: the flags of England, Scotland and Wales are U+1F3F4, a run of tag characters, and U+E007F (Wikipedia). Strip the whole block and a Welsh customer's flag becomes a plain black flag. And the zero-width joiner and non-joiner that the book strips are spelling in places. ZWNJ is required in Persian words like می‌خواهم and in several Indic scripts, and ZWJ is what makes 👩‍💻 one emoji rather than two.

python
_RGI_FLAGS = ("gbeng", "gbsct", "gbwls")   # the only RGI subdivision flags_FLAG = re.compile("|".join(    "\U0001F3F4" + "".join(chr(0xE0000 + ord(c)) for c in code) + "\U000E007F"    for code in _RGI_FLAGS))_TAG = re.compile("[\U000E0000-\U000E007F]")_INVISIBLE = re.compile("[\x00-\x08\x0b\x0c\x0e-\x1f\x7f\u200b\u200e\u200f\u2028\u2029"                        "\u2060\u00ad\ufeff"                        "\u202a-\u202e\u2066-\u2069]")def _joiner_has_a_job(prev: str, nxt: str) -> bool:    """ZWJ and ZWNJ are spelling in Persian, Indic scripts and emoji sequences."""    def scripted(c: str) -> bool:        cat = unicodedata.category(c)        return cat in ("So", "Sk") or (ord(c) > 0x036F and cat[0] in "LM")    return scripted(prev) or scripted(nxt)def strip_invisible(text: str) -> str:    keep = {m.span() for m in _FLAG.finditer(text)}    out = []    for i, c in enumerate(text):        inside_flag = any(a <= i < b for a, b in keep)        if _TAG.match(c) and not inside_flag:            continue        if _INVISIBLE.match(c):            continue        if c in "\u200c\u200d":            prev = text[i - 1] if i else " "            nxt = text[i + 1] if i + 1 < len(text) else " "            if not _joiner_has_a_job(prev, nxt):                continue        out.append(c)    return "".join(out)

Run through this sensor (strip-invisible, NFKC, then strip-invisible again as strip-after-nfkc), the inputs from the table above and five more give:

text
input              len delta  removed inserted  evidencefullwidth refund           0        6        6  {'compatibility-form': 6}ellipsis                  -2        1        3  {'compatibility-form': 1}tag smuggling             11       11        0  {'tag': 11}fake flag wrapper         12       12        0  {'tag': 12}Wales flag                 0        0        0  {'tag': 6}emoji ZWJ                  0        0        0  {'joiner': 1}Persian ZWNJ               0        0        0  {'joiner': 1}fullwidth ZWJ              1        7        6  {'compatibility-form': 6, 'joiner': 1}ZWJ inside refund          1        1        0  {'joiner': 1}fullwidth ZWJ after all three steps: 'refund'

Now the tag payload is gone and counted, including when it is wrapped in a fake flag: only the three RGI subdivision flags (England, Scotland, Wales) are kept whole, so U+1F3F4 plus a tag-encoded payload plus U+E007F loses all 12 tag characters. Flag, emoji and Persian word all come through untouched, and the evidence column still notes what they contained, because evidence describes the raw text and does not judge it. A joiner between two Latin letters, as in re, a ZWJ, fund, has no orthographic job and is removed. The second pass exists for the fullwidth ZWJ row: a joiner between fullwidth letters survives the first pass, because fullwidth letters sit above U+036F, and NFKC then folds them into Latin, leaving r, a ZWJ, efund. Stripping again after NFKC turns it into refund, and the first pass still runs before NFKC, so the cafe case above keeps its order. This joiner rule is a heuristic with a known edge: it decides from neighbouring characters, not from a script property, and a language team will find cases it gets wrong. That is acceptable for a sensor whose every change is counted, and it would not be acceptable for a gate.

d2
grid-rows: 1
grid-gap: 30
strip: "Stripped and counted" {
  label.near: top-center
  style: {fill: "#FDF2E9"; stroke: "#D9785A"}
  grid-columns: 1
  grid-gap: 14
  a1: "tag characters outside a flag" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
  a2: "bidi controls U+202A-E, U+2066-9" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
  a3: "ZWSP, word joiner, soft hyphen, BOM" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
  a4: "a joiner between Latin letters" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
}
keep: "Kept, noted as evidence" {
  label.near: top-center
  style: {fill: "#EAF7F0"; stroke: "#3F9E52"}
  grid-columns: 1
  grid-gap: 14
  b1: "tags inside a subdivision flag" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
  b2: "ZWJ inside an emoji sequence" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
  b3: "ZWNJ in Persian or Indic text" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
  b4: "Cyrillic homoglyphs: UTS #39's job" {style: {fill: "#F4F6F7"; stroke: "#95A5A6"; font-color: "#2C2C2A"}}
}

The grey cell on the right is a deliberate gap. NFKC does not fold a Cyrillic о into a Latin o, because they are not compatibility-equivalent. Confusable detection is a separate mechanism with its own data, Unicode's UTS #39 (Unicode), and bolting it into the strip step would merge two jobs again. Variation selectors are an uncovered gap of a different kind. U+FE00 to U+FE0F and U+E0100 to U+E01EF have no visible glyph of their own, and a string carrying them passes both evidence and strip_invisible unchanged in this code. Add them deliberately if your traffic has no emoji presentation needs, and count them either way.

Decoding is an admission decision, and your framework may have made it already

Everything above assumed a string. Something made that string from bytes, using a charset from a header, a meta tag or a default, and any of the three can be wrong.

On this the book is right: a decoding failure is a refusal ("this cannot be processed as text"), so it belongs in the gate. Handling it with errors="replace" inside the normaliser puts a refusal inside the component that is not allowed to refuse. Where I part company with the book is its claim that strict decoding is one line of configuration in most frameworks. Check what yours does. Here is Starlette's multipart form parser on its master branch, which FastAPI uses for form fields:

python
def _user_safe_decode(src: bytes | bytearray, codec: str) -> str:    try:        return src.decode(codec)    except (UnicodeDecodeError, LookupError):        return src.decode("latin-1")

There is no setting. JSON bodies take a different path, through json.loads, which raises UnicodeDecodeError on invalid UTF-8 instead of falling back. The fallback is on the form path, which is the one that matters for email: inbound-mail webhooks such as SendGrid's Inbound Parse post multipart/form-data (SendGrid). A field with one bad byte is re-decoded as Latin-1, which never fails, so every valid multi-byte character in the same field becomes mojibake too. Werkzeug, under Flask, removed its encoding_errors attribute in 3.0, and get_data(as_text=True) decodes with errors="replace", Python's replacement-marker handler. In Part 3's terms the body parser looks like Harness Residual, plain plumbing reviewed as application code. It is not. It is an unrecorded sensor sitting where a gate belongs, transforming bytes it should have been allowed to refuse. Here are 24 bytes of valid UTF-8 followed by one stray 0xFF byte, decoded three ways:

text
latin-1 fallback: '£940 refund, ref café ÿ'errors=replace:   '£940 refund, ref café �'NotAdmitted: ticket_body: not utf-8 at byte 24

Of the three, the Latin-1 line is the worst outcome on the page: the pound sign is now two characters, the amount a parser pulls from that text is not £940, and there is no U+FFFD for anything to count. The replacement-character line is defensible, and the book is right to count it rather than ban it: Unicode's security report recommends substituting U+FFFD over silently skipping bytes (UTR #36). Its problem is that it is silent and uncounted, so if your business must accept undecodable input, count the U+FFFD characters as evidence, and treat a rise as a fact about an upstream system. Make the third line your default: read the raw bytes yourself and decode them in the gate.

python
CHARSETS = frozenset({"utf-8", "cp1252", "iso8859-1", "shift_jis"})  # codecs.lookup namesRULES = {    "system_prompt": FieldRule("operator", 4_000, 16_000),    "ticket_body": FieldRule("customer", 8_000, 32_000),    "invoice_row": FieldRule("tool-result", 2_000, 8_000),}@dataclass(frozen=True)class Admitted:    """Built only by admit(). The sensor reads .text and never decodes again."""    text: Mapping[str, str]    charset: Mapping[str, str]def admit(raw: Mapping[str, tuple[bytes, str]]) -> Admitted:    """Each field arrives as (bytes, declared charset). Refuse, or decode exactly once."""    text, used = {}, {}    for name, (body, charset) in raw.items():        rule = RULES.get(name)        if rule is None:            raise NotAdmitted(f"{name}: undeclared field; add it or stop sending it")        if len(body) > rule.max_bytes:            raise NotAdmitted(f"{name}: {len(body)} bytes, limit {rule.max_bytes}")        try:            codec = codecs.lookup(charset).name     # windows-1252 -> cp1252        except LookupError:            raise NotAdmitted(f"{name}: unknown charset {charset!r}") from None        if codec not in CHARSETS:                   # utf-7, unicode_escape, idna...            raise NotAdmitted(f"{name}: charset {charset!r} not allowed")        try:            decoded = body.decode(codec)            # strict is the default        except UnicodeDecodeError as exc:            raise NotAdmitted(f"{name}: not {charset} at byte {exc.start}") from None        if len(decoded) > rule.max_chars:            raise NotAdmitted(                f"{name}: {len(decoded)} characters, limit {rule.max_chars}"            )        text[name], used[name] = decoded, codec    return Admitted(text, used)
text
NotAdmitted: crm_note: undeclared field; add it or stop sending itNotAdmitted: ticket_body: 9000 characters, limit 8000admitted: ticket_body 11 characters, 11 bytes, cp1252admitted: ticket_body 7000 characters, 21000 bytes, utf-8NotAdmitted: ticket_body: charset 'utf-7' not allowed

Here I part from the book, whose admit checks strings the framework already decoded and returns None. That leaves the one decode with the framework, which on Starlette's form path is the Latin-1 fallback above. Decoding is the one conversion that belongs to the gate, because its failure is a refusal, so this admit takes bytes and returns an Admitted holding the decoded text and the charset it used for each field. Each field is decoded in the charset its MIME part declared, because real mail arrives in windows-1252 and Shift_JIS as well as UTF-8, and only from an allowlist, because Python will also decode utf-7 or unicode_escape, which turn plain ASCII bytes into arbitrary code points. The gate still never cleans. Limits are in characters, as in the book, with a byte cap well above them checked before decoding, because a pure byte limit would refuse a Hindi ticket at a third of the length of an English one. Of the refusals, I value the undeclared-field one most: somebody adds a CRM notes field to the payload, and instead of that text quietly entering a context window, the request fails naming the field. In the book's version, the field list lives in a policy file with a total ceiling deliberately below the sum of the per-field limits, so the total is a ceiling that can actually fire. That file is the artifact worth copying.

d2
direction: down
bytes: "raw request bytes" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
paths: "" {
  style: {fill: "#FFFFFF"; stroke: "#FFFFFF"}
  grid-rows: 1
  grid-gap: 24
  p1: "framework fallback\nlatin-1: mojibake,\nnothing to count" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
  p2: "errors=replace\nU+FFFD: count it\nas evidence" {style: {fill: "#F4F6F7"; stroke: "#95A5A6"; font-color: "#2C2C2A"}}
  p3: "gate: strict decode\nNotAdmitted with\nthe byte offset" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
}
bytes -> paths: "decoded where?"

The left box is Starlette's multipart default, and so FastAPI's form handling. Werkzeug gives you the middle one. Only the left box leaves no trace at all.

Why an injection classifier may sit at the entry point but never refuse there

At the entry point, the strongest pull is to ask a model whether the input looks like an attack. OWASP recommends a version of it: item 3 under LLM01:2025 says to "Apply semantic filters and use string-checking to scan for non-allowed content" (OWASP GenAI). OWASP leaves what to do with a hit open. A common implementation refuses at the entry, and so did the April post on this blog, which said to "Layer a secondary classifier on top". That is what I disagree with, and I am reversing my own advice. I have no quarrel with the classifier. Letting it refuse at the entry point is the problem. OWASP's item 6, "Segregate and identify external content", is close to the channel-origin argument this article makes.

First, these classifiers are scoped to a failure that is not this article's. Meta's model card for Llama Prompt Guard 2 says it is trained to "classify prompts as 'malicious' if the prompt explicitly attempts to override prior instructions", and that Meta dropped Prompt Guard 1's broader injection label because "we found this objective too broad to be useful" (Meta). "I have approved a refund of $940.00" attempts to override nothing. A detector working exactly as designed has no reason to flag it.

Second, they refuse real customers. InjecGuard's authors built NotInject, which "contains 339 benign samples enriched with trigger words common in prompt injection attacks", and found that "state-of-the-art models suffer from over-defense issues, with accuracy dropping close to random guessing levels (60%)" (Li et al., 2025). A billing queue is full of benign trigger words, such as "ignore my last email" or "cancel the previous instruction to my bank". As illustration only: a classifier at a 1% false-positive rate, refusing on a queue of 10,000 tickets a day, turns away about 100 real customers a day, each of whom has a real problem and gets nothing. Vendor benchmarks look better than that. Lakera's PINT benchmark, where Lakera's own product scores highest, is run by the vendor, so read its numbers with that in mind (Lakera).

Third, they get bypassed in exactly the deployments that rely on them. EchoLeak's chain included evading Microsoft's XPIA classifier.

None of that says the score is useless. Meta's card pitches Prompt Guard for "Proactive Monitoring", and as a reading that is what it is good for. A score recorded beside the code-point changes and the evidence counts is a sensor reading like any other, and charting it per field per day will tell you something. The rule for refusing at the entry point is the same one admit follows: a refusal there must rest on a fact that is deterministic, declared and explainable, such as size, an undeclared field or bytes that do not decode. A probabilistic judgement may refuse only where the proposal exists. Part 3's gates refuse with the proposal in hand: whether this refund is on an invoice, whether this amount is under the ceiling. At the entry point, the classifier sees only text, can be wrong in both directions, and the refusing direction lands on a customer.

d2
grid-rows: 1
grid-gap: 34
asgate: "Classifier as gate at the entry" {
  label.near: top-center
  style: {fill: "#FDF2E9"; stroke: "#D9785A"}
  grid-columns: 1
  grid-gap: 14
  g1: "sees text only, no proposal" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
  g2: "false positive: customer refused" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
  g3: "laundered approval: not flagged, passes" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
}
assensor: "Classifier as sensor at the entry" {
  label.near: top-center
  style: {fill: "#EAF7F0"; stroke: "#3F9E52"}
  grid-columns: 1
  grid-gap: 14
  s1: "score written to the record" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
  s2: "false positive: a chart blip" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
  s3: "laundered approval: not flagged, recorded" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
  s4: "refusals at the tool gate and reply check" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
}

Both sides share one fact in the third row: neither classifier flags the laundered approval. What catches it is the channel check above, and two checks downstream of the model: a refund gate that reads the ledger before money moves, and, because the reply is an action too, a check of what the reply claims against the ledger before it is sent. This series has not built that reply check yet, so do not assume Part 3's refund gate covers it. Nothing at the entry point can play either role. Using the classifier as a gate, the left column, adds a failure mode that lands on customers, and the failures it does remove are better caught where the output or the proposal exists.

Spotlighting deserves a qualification here. It marks untrusted spans so the model can tell them apart, and it cut attack success "from greater than 50% to below 2%" in its authors' experiments on GPT-family models (Hines et al., 2024). That is a transformation, so it belongs in the sensor, and it is a probability shift inside the model rather than a control, for the reason Part 2 gave. Use it, and do not count it as a gate.

Storing raw LLM input under GDPR: keep it in the system of record

Anomaly evidence is only as good as your ability to go back to the bytes. A postmortem where the only surviving copy of the input is the cleaned one cannot say what arrived. So keep the original, and keep it out of the context window.

That runs straight into data protection. A ticket body is personal data, and GDPR Article 5 requires it to be "adequate, relevant and limited to what is necessary" and "kept in a form which permits identification of data subjects for no longer than is necessary" (GDPR Art. 5). An incident store holding a second copy of every raw ticket, next to the cleaned copy, gives you two copies of the problem and a retention schedule nobody wrote.

My rule: the raw ticket lives in the ticketing system that already holds it, under the retention policy that already governs it, and the sensor's record points to it by ticket ID. Inside the record are the field name, the step, the code-point counts, the evidence counts, the classifier score and a keyed HMAC of the exact bytes admit received, so you can prove two incidents saw the same bytes without storing them twice. Check whether your ticketing system keeps those bytes or its own sanitised copy. Many keep the copy, and then evidence can only be recomputed from what you hashed and counted at the time. For fields whose content is the PII itself, such as get_account output, the book's inversion applies: keep the hash and the count, never the value.

d2
grid-rows: 1
grid-gap: 30
sor: "Ticketing system of record" {
  label.near: top-center
  style: {fill: "#EAF7F0"; stroke: "#3F9E52"}
  grid-columns: 1
  grid-gap: 14
  t1: "raw ticket body" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
  t2: "existing retention policy" {style: {fill: "#F4F6F7"; stroke: "#95A5A6"; font-color: "#2C2C2A"}}
}
rec: "Sensor record" {
  label.near: top-center
  style: {fill: "#EEF3FB"; stroke: "#2C6FB0"}
  grid-columns: 1
  grid-gap: 14
  r1: "ticket ID + HMAC of raw bytes" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  r2: "changes, evidence, score" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
}
ctx: "Context window" {
  label.near: top-center
  style: {fill: "#FDF2E9"; stroke: "#D9785A"}
  grid-columns: 1
  grid-gap: 14
  c1: "canonical text only" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
  c2: "no raw copy; tool PII redacted" {style: {fill: "#F4F6F7"; stroke: "#95A5A6"; font-color: "#2C2C2A"}}
}

Three places, one copy of the raw text. The context window still holds the canonical ticket, which is personal data on its way to a provider. Only the middle box is new storage, and it holds counts and an HMAC, not customer words, though under GDPR a ticket ID plus a hash is still pseudonymous personal data and takes the ticket's retention period.

A record nobody reads is a log line, which brings up the honest state of the book's own code. Chapter 4's graph node calls the sensor, binds its removals, and drops them, because nothing in the system has a place to put a sensor's record until Chapter 6. That is a deliberate stage in a book that builds the harness one component at a time. In your system, ship the record and its destination together. The number worth charting is code points changed per field per step per day. It is also the opposite of what Part 1 called Borrowed Evidence: a measurement of your harness, taken in your own domain, instead of a result carried over from coding agents. A step whose changes jump tenfold has started catching something or started breaking something. A step whose changes drop to zero after a non-zero baseline has probably stopped working, and nothing else will tell you, because a sensor that always succeeds never produces an alert. Rare-event steps such as tag stripping record zero most days, so feed them a daily canary input, one tag character and one fullwidth word, to give them a baseline.

What would change this

Two findings would move me. If a published incident showed a refusal-capable entry classifier stopping a laundered-provenance failure that the downstream tool gate would have let through, the "never refuse at the entry" rule would need an exception. And if role-typed serialisation, the property that took compaction laundering to 0% in Dabirsiaghi's measurements, were extended to every hop a reply takes, from your outbound mail to the inbound ticket, the In-Reply-To check would become a special case of something more general, which I would prefer.

Checklist: audit your LLM input validation by its record

  1. List your input validation's verbs in two columns. Transforming verbs go to a sensor that cannot raise. Refusing verbs go to a gate that returns only the decoded text. Tests that fail after the split were testing the merge.
  2. Run the gate on raw bytes. Size limits, declared fields and strict decoding in each part's declared charset happen before any transformation, and the sensor reads the gate's one decode. Find out what your framework does with an undecodable byte, and if it is Starlette's multipart path or Werkzeug's get_data, decode the raw body yourself.
  3. Record changes in code points, not length. Removed and inserted per field per step, so a fullwidth fold and an ellipsis expansion both show up.
  4. Read evidence before the first step. Tag characters, bidi controls, invisible characters, joiners, U+FFFD and compatibility forms, counted on the raw text.
  5. Strip the tag block except the three RGI subdivision flags, and keep joiners that have a job. Add a test with 👩‍💻, a Welsh flag, a Persian word and a tag payload wrapped in a fake flag, so a future tightening of the strip fails CI instead of a customer.
  6. Assign origin from the channel. For email, match In-Reply-To against your outbound store and record own-quote or quote-mismatch. Find every field more than one subsystem writes to, and split it.
  7. Pin the step order with a test per load-bearing pair. Start with the cafe plus U+200D plus U+0301 case.
  8. Refuse at the entry only on declared, deterministic facts. A classifier score goes into the record. A refusal based on it belongs at the gate that can see the proposal.
  9. Keep the raw text in the system of record. Your sensor record holds a ticket ID and a hash, not a second copy.
  10. Chart changes per field per day, and alert when a step drops to zero. Give rare-event steps a daily canary input so they have a baseline to drop from.

Going deeper: Chapter 4 of the book

This is Part 4 of a series that follows Harness Engineering for Production AI Systems one chapter at a time. Chapter 4, "Input Defense and Normalisation," builds the declarative input policy and its tests, redaction as a sensor with its own ordering trap, why NFKC over the other three normalisation forms, the admission gate that does not fit Chapter 3's gate protocol, and a measured finding about what survives a LangGraph node boundary that changes how every later chapter's nodes are written. Code for the chapter is the ch04-input-defense tag of the companion repository, and its tests need no model and no API key.

References


AI Security

Agentic AI

Follow for more technical deep dives on AI/ML systems, production engineering, and building real-world applications:


Books by Ranjan Kumar

Federated AI Agents cover

Federated AI Agents

Right-Sized AI cover

Right-Sized AI

Harness Engineering for Production AI Systems cover

Harness Engineering

The 7 GenAI Architectures cover

The 7 GenAI Architectures

Building Real-World Agentic AI Systems with LangGraph cover

Building Real-World Agentic AI Systems

The ChatML Handbook cover

The ChatML Handbook

The Chat Templates Handbook cover

The Chat Templates Handbook

Comments