HarnessDev: A Model Can Write What Its Executor Reads

Claude Opus 4.8 built itself a coding harness and scored 69.3 on SWE-bench Pro. That harness was then frozen. A different model was put behind it, and the score fell to 33.0. On the paper's own clean-sensitivity basis, 49.1.

Nothing in the harness changed. No prompt was rewritten and no tool was removed. Somewhere in its evolution the harness had written down a constant, and the paper names the mechanism in one of Opus's code harnesses: a hard-coded 120-step limit, fitted around the executor that happened to be running at the time.

That is a good limit. It is a real observation about how Opus works, learned from real runs and encoded so the harness stops burning tokens on a task that has already stopped converging. It is also a sentence about Opus written into a file that Opus is not guaranteed to be reading next quarter. That collapse is HarnessDev's headline result, and the benchmark behind it is the first to grade a model on building its own harness rather than on working inside one.

Current as of 2026-09-21, against arXiv:2609.01437v1 (submitted 1 September 2026) and claude plugin eval as it ships in Claude Code v2.1.269.

The thesis: a model can write what its executor reads, not what binds it

The obvious reading of HarnessDev is a verdict on self-evolving harnesses. That is broader than what the paper measured, and it is the less useful reading.

A model can safely author the part of its own harness that the executor is free to ignore. It cannot safely author the part that binds the executor.

Prompts, skills, and memory notes are read, and the executor takes them as input and decides what to do. Step budgets, stopping rules, message protocols, and context compression are obeyed. None of those is up for negotiation. By the time they apply, a decision has already been taken on the executor's behalf, using an assumption about how it behaves.

That assumption is the whole problem. A model that writes its own advisory material is writing down what it knows about the task. A model that writes its own binding logic is writing down what it knows about itself, and that is the part which does not survive being handed to a different model.

Read and obey is the operational test, not the underlying variable. What decides transfer is how much room the artifact leaves the writing model to encode an assumption about itself, and the line leaks in both directions. Free-prose memory is read-layer by form and still moves accuracy by 13.28 points under a writer swap, because compressing a history into prose is an invitation to encode yourself. A step budget of 10,000 sits in the obey layer and binds nothing, so it carries no coupling and survives any swap. The proxy holds because the two usually travel together: a constant is worth setting only when it is tight enough to fire, and a constant tight enough to fire was fitted to somebody.

Self-evolving harnesses do not fail. But "let the agent evolve its own harness" is two decisions with opposite answers, and almost nobody is splitting them. On a brownfield codebase the harness is already the product. HarnessDev is the first measurement of what happens when the product starts writing itself.

Why HarnessDev and Lin et al. reached opposite conclusions

In May 2026, Lin et al. published Harness Updating Is Not Harness Benefit. They separated the model that writes harness updates from the model that executes under them, then asked how much the writer's identity matters.

Barely at all. Across benchmarks, best and worst evolver were separated by at most 3.1 percentage points, and no model won everywhere. In one case a Qwen3.5-9B evolver produced a skill the authors describe as "procedurally isomorphic" to the one Claude Opus 4.6 wrote for the same task. The two differed only in surface detail, and both lifted the same Opus agent from 0.67 to 1.0 on a SkillsBench task. Their recommendation was blunt: allocate capability budget to the task-solving agent, not to the evolver.

Four months later HarnessDev swaps the executor and 36 points disappear.

Both results are correct. They are measurements of different layers. Lin et al. held tool interfaces and execution policies fixed and evolved prompts, skills, and memories. HarnessDev hands the model a seed harness that scores zero on every benchmark unmodified: no agent loop, no tool policy, no context management, no state, no verifier, no retry path, no stopping rule. Then it asks for the whole execution stack, written as code.

So the two papers bracket the line rather than contradict each other. Everything Lin et al. evolved is advisory and transferred fine. Everything HarnessDev added is binding, and every coupling instance the paper names sits in exactly that added layer: the 120-step limit, a sanitizer that breaks valid Gemini tool-result sequences, context compression that breaks tool-message pairing, and deduplication and termination rules the paper says are "adapted to the original model."

A third result lands on the same line from a different direction. Goyal and Ray ran a controlled study of whether an agent's memory survives a model upgrade, published 4 September 2026. Model-compressed prose notes shifted accuracy asymmetrically by +9.91 or -13.28 percentage points depending on which direction the migration ran. Fixed-schema knowledge-graph entries shifted by 0.0004 plus or minus 0.0020 in raw accuracy after the same writer swap, which is four hundredths of a percentage point. Same artifact class, same task, and the only difference is whether the format left the writing model room to encode itself. Their study is small and synthetic, at 48 generated histories across two open-weight sub-10B models, so read it as direction rather than magnitude.

What HarnessDev actually measured

HarnessDev is the first benchmark that grades a model on building and evolving its own harness rather than on completing tasks inside one someone else built. It comes from a 19-author team led from ByteDance Seed, submitted 1 September 2026 as arXiv 2609.01437.

The scale is real: 2,207 unique downstream instances across five benchmarks in four domains. SWE-bench Pro public split at 731 instances and Terminal-Bench 2.1 at 89 cover code, MLE-bench at 75 covers machine-learning experimentation, EQ-Bench3 at 46 covers writing, and BrowseComp at 1,266 covers search and research. Six creator models, three independent harnesses per creator-benchmark pair, reported as the mean of three runs.

Two evaluation views matter. Self-Eval is the creator running as its own executor. Unified-Eval freezes each creator's harness and runs every one of them under a single fixed executor, Gemini 3.1 Pro. That gap is the whole experiment.

One scoping fact has to be stated before anything else, because it is in the abstract and any reader is one click from it. Generated harnesses trail mature human-engineered references on code and on search and research, while matching or exceeding the selected references on writing and machine-learning experimentation.

Reference values and the BrowseComp figure come from MarkTechPost's coverage; the paper's tables are typeset as math and did not survive text extraction, though its prose confirms the direction of every row. Treat the references carefully in either case. Figure 4's own caption says they "come from different harness-model combinations and are not paired controls under a common executor," and three of them are lifted from OpenAI's GPT-5.6 report rather than re-run. "Exceeding" means exceeding a published system, not exceeding human ability.

Read against the thesis, that table stops being an inconvenience. Writing and machine-learning experimentation are thin-harness domains. In both, the model's output is the product and the harness mostly stays out of the way. Code is thicker: there is a loop, a step budget, a completion check. Search and research is the thickest of all, because the harness has to carry query deduplication, result review, and a termination rule across a long horizon, and it is the domain where a model-written harness is furthest behind by a wide margin.

The more control logic a harness carries, the worse a model does at writing it. That ordering is consistent with the read/obey split. It is not a test of it, and three things stop it being one. The Gap column is measured against references I disclaimed a paragraph ago as unpaired and externally sourced. Point gaps across benchmarks with different scales and ceilings are not commensurable, and a rubric-scored writing benchmark is a compressed scale in a way BrowseComp is not. Most of all, there is no executor swap anywhere in that table, and transfer under a swap is the entire claim. A duller hypothesis fits the same ordering: a thicker harness is a harder engineering problem, so a mature reference system beats a few rounds of evolution by more.

The strongest evidence is the layer contrast instead. Lin et al. froze the executable layer and found the evolver worth at most 3.1 points. HarnessDev unlocked it and found 36. Two studies, the same question, one variable between them, and that is the comparison the argument should rest on.

Author Premium: a number you can compute for your own harness

Author Premium is the signed change in a harness's score when the model executing under it changes and nothing else does:

Author Premium = score(author model + its own harness)               - score(different executor + the same frozen harness)

Positive is the bad direction, which is worth saying plainly because the word premium suggests otherwise. A positive premium means the harness was co-adapted to the model that wrote it, so the gain is a liability you pay out the day you switch. Negative means the author model was the bottleneck and the harness is carrying compensations for weaknesses the new executor does not have. Either way, the sign is the finding.

There is also no single premium for a harness. It is a property of three things, not one: the harness, the model that authored it, and the specific executor you compare against. Opus against Gemini and Opus against Haiku are different measurements, so a premium is only meaningful when it names the comparison executor. Agent harness portability is not a property a component has or lacks. It is a number against a named alternative.

HarnessDev gives four measurable instances:

The Qwen rows carry only a delta because the paper reports those two as movements under the swap rather than as a pair of scores. Both Opus rows you can check by subtraction.

The parenthetical matters, and the coverage dropped it. Table 4's caption reports that some code cells contain one collapsed replica out of three, and that dropping it gives a clean-sensitivity mean of 49.1 for Opus on SWE-bench Pro rather than 33.0. So the honest headline is 69.3 to 33.0 as reported, or 69.3 to 49.1 on the authors' own sensitivity basis. Cite both. A 20-point premium is still the largest measured effect in the paper, and quoting only the worse number invites a reader to check Table 4 and stop trusting the rest.

One row cuts against the thesis, and burying it would be the same error the previous paragraph just warned about. Writing is a thin-harness domain, so an EQ-Bench3 harness should be mostly advisory and should show a premium near zero. It shows +10.4. Two readings survive and I cannot separate them from the published numbers. Thin does not mean none: an EQ-Bench3 harness still carries a stopping rule and a context policy, so the prediction is an ordering of magnitudes rather than a floor at zero, and 10.4 against the code row's 20.2 is the right order. The other reading is that a rubric-scored writing benchmark measures executor writing quality more than harness transfer, in which case the same doubt applies to reading the code number. The writing row is weak evidence, in both directions.

An obvious objection: Gemini 3.1 Pro might simply be a weaker SWE-bench Pro executor than Opus 4.8, which would make this a model-quality result wearing a harness costume. HarnessDev concedes the confound directly: Self-Eval "reflects harness design, executor capability, and the compatibility between them."

The mechanism answers it. The paper names two couplings that are harness properties rather than model properties: the hard-coded step limit, and a duplicate-query rate on the Opus search harness that MarkTechPost reports rising from 10.1% to 88.2% once the executor changed. A deduplication rule that suppresses nine out of ten redundant queries for one model and one in ten for another is not measuring model quality. It is a rule fitted to one model's query-generation behaviour, and a model swap walks straight past it.

The sign reversal looks like a second answer and I am not going to use it as one. Qwen's harness went up 17.6 points on BrowseComp and 12.9 on MLE-bench under Gemini, which reads as proof that the effect cannot be executor capability. It is not, because executor capability is benchmark-specific. Gemini being a worse SWE-bench Pro executor than Opus and a better BrowseComp executor than Qwen is an ordinary state of affairs, and it reproduces the entire pattern with no coupling at all. The comparison changes creator, benchmark and direction at once, so it isolates nothing. What would settle it is a sign reversal across creators within one benchmark under the same fixed executor. The paper does not report one.

One thing I cannot claim, and neither can anyone else yet: HarnessDev never ran the swap in reverse. Gemini 3.1 Pro is the only fixed executor in the paper. There is no arm where a Gemini-authored harness runs under Opus.

Where Jev Belongs in an Agent Harness separates two ways a fitted number goes wrong: drift is the model changing underneath a threshold, a transplant is the population changing. A frozen step budget under a new executor is both at once. The model changes, and the distribution of trajectory lengths under that constant changes with it. That is why a swap is worse than either failure alone, and why an executor rollout deserves the scrutiny a population shift gets.

Advisory vs binding: where the line falls in your harness

direction: right

read: "READ layer: what the model wrote that the executor may ignore" {
  style: {
    fill: "#95A5A6"
    stroke: "#6E7F80"
    font-color: "#2C2C2A"
  }
  r1: "System prompt\nand instructions" {
    style: { fill: "#98D8C8"; stroke: "#4E9182"; font-color: "#2C2C2A" }
  }
  r2: "Skills and\nSKILL.md files" {
    style: { fill: "#98D8C8"; stroke: "#4E9182"; font-color: "#2C2C2A" }
  }
  r3: "Memory entries\nin a fixed schema" {
    style: { fill: "#98D8C8"; stroke: "#4E9182"; font-color: "#2C2C2A" }
  }
}

obey: "OBEY layer: what the model wrote that binds the executor" {
  style: {
    fill: "#95A5A6"
    stroke: "#6E7F80"
    font-color: "#2C2C2A"
  }
  o1: "Step and\nturn budgets" {
    style: { fill: "#E74C3C"; stroke: "#A93226"; font-color: "#FFFFFF" }
  }
  o2: "Stopping and\ntermination rules" {
    style: { fill: "#E74C3C"; stroke: "#A93226"; font-color: "#FFFFFF" }
  }
  o3: "Message and\ntool-result protocol" {
    style: { fill: "#E74C3C"; stroke: "#A93226"; font-color: "#FFFFFF" }
  }
  o4: "Context\ncompression policy" {
    style: { fill: "#E74C3C"; stroke: "#A93226"; font-color: "#FFFFFF" }
  }
}

swap: "Freeze the harness.\nChange only the executor." {
  style: {
    fill: "#FFD93D"
    stroke: "#C9A400"
    font-color: "#2C2C2A"
  }
}

ported: "Survives the swap\n\nat most 3.1 pt spread across\n7 evolvers (Lin et al.)\n\n+0.0004 for fixed-schema memory\nafter a writer swap (Goyal and Ray)" {
  style: {
    fill: "#6BCF7F"
    stroke: "#3F9E55"
    font-color: "#2C2C2A"
  }
}

broke: "Does not survive the swap\n\nSWE-bench Pro 69.3 -> 33.0\n(49.1 on the clean basis)\n\nduplicate-query rate\n10.1% -> 88.2%" {
  style: {
    fill: "#C2185B"
    stroke: "#8C1044"
    font-color: "#FFFFFF"
  }
}

read -> swap: "advisory" { style: { stroke: "#4E9182"; font-color: "#2C2C2A" } }
obey -> swap: "binding" { style: { stroke: "#A93226"; font-color: "#2C2C2A" } }
swap -> ported { style: { stroke: "#3F9E55" } }
swap -> broke { style: { stroke: "#8C1044" } }

That line is not drawn by artifact type. It is drawn by whether the executor retains the decision. A skill that says "prefer integration tests over unit tests for this repo" leaves the executor free to disagree, and a different model reading it will do something sensible. A stopping rule that returns after 120 steps has already decided, using a number that was true of one model on one workload. That is the oracle problem with a constant standing in for the oracle.

The wrong way: shipping the edit because the score went up

The paper names the 120-step limit as a coupling instance but does not print the code, so the block below is my reconstruction of the shape such an edit takes. The plausible provenance is that the agent watched runs past roughly 120 steps fail to converge, and wrote down what it saw.

# harness/loop.py - illustrative reconstruction of the coupling the paper names.from harness.runtime import Timeout, executeMAX_STEPS = 120def run(task, executor):    for _ in range(MAX_STEPS):        action = executor.next_action(task)        if action.kind == "finish":            return action.result        task.observe(execute(action))    raise Timeout(f"gave up after {MAX_STEPS} steps")

The score went up. It is a small change, and the evidence behind it is real: the agent collected those runs itself. A reviewer skimming the diff sees a sensible timeout and approves it.

Moving the constant to a configuration file or an environment variable does not fix this. MAX_STEPS=120 in YAML is the same sentence about Opus, stored somewhere else. The defect is not that the value is hard-coded. It is that the value was fitted to one model and nothing in the system records which one.

Failure only appears when the executor changes. A model that plans in smaller increments hits step 120 in the middle of a task rather than at the end of a doomed one, and every one of those truncations is recorded as a task failure. The harness reports a model regression. The regression is in the harness.

The right way: make the binding a property of the pairing

A binding constant that was measured against one model belongs in a table keyed by that model, not in a module-level constant. And an unmeasured pairing should refuse to run rather than guess.

# harness/budgets.py - generated by the swap suite; do not hand-edit.# (dated model id, workload class) -> (steps, measured on, harness version)MEASURED_STEP_BUDGETS = {    ("opus-4.8-20260901", "long-refactor"): (120, "2026-09-10", "harness-3.2"),    ("gemini-3.1-pro-20260815", "long-refactor"): (300, "2026-09-10", "harness-3.2"),}class UnmeasuredPairing(RuntimeError):    passdef budget_for(model_id, workload, harness_version):    try:        steps, measured_on, measured_against = MEASURED_STEP_BUDGETS[            (model_id, workload)        ]    except KeyError:        raise UnmeasuredPairing(            f"no measured budget for {model_id} on {workload}"        ) from None    if measured_against != harness_version:        raise UnmeasuredPairing(            f"budget for {model_id} on {workload} was measured {measured_on} "            f"against {measured_against}, not {harness_version}"        )    return stepsdef validate_pairings(model_ids, workloads, harness_version):    """Call at boot, so an unmeasured rollout fails the deploy, not a request."""    for model_id in model_ids:        for workload in workloads:            budget_for(model_id, workload, harness_version)

# harness/loop.pyfrom harness.budgets import budget_forfrom harness.runtime import Timeout, executedef run(task, executor, harness_version):    limit = budget_for(executor.model_id, task.workload, harness_version)    for _ in range(limit):        action = executor.next_action(task)        if action.kind == "finish":            return action.result        task.observe(execute(action))    raise Timeout(f"gave up after {limit} steps")

Those numbers are yours, not the paper's. What matters is the shape. The constant is now visibly a fact about a pairing rather than about the harness, validate_pairings turns an unmeasured rollout into a failed deploy rather than a slow bleed on live traffic, and the harness version is in the key because a budget measured against harness 3.2 is as stale on harness 4.0 as it would be on a different model. The diff that adds a row is the diff a reviewer should be reading.

Failing closed is not free, and an article that recommends it owes you the bill. A missing row takes a whole pairing to zero throughput. That is the right trade where an overrun is expensive, on long autonomous runs that spend real money before anyone notices. It is the wrong trade on an interactive path, where the better failure is to fall back to the last measured model, serve the request, and page someone. What you must not do is let an unmeasured pairing run on a guessed number, because that is the original defect with extra steps.

This is the same move as putting enforcement in code rather than in a prompt, which is the argument in Hooks: The Enforcement Layer That Turns Agent Policy Into Agent Fact. The axis itself is not new here. Skills vs Hooks in Claude Code named it the Enforceability Axis, with advisory and enforced as its two poles, and asked which tool a given rule belongs in. This article asks a different question on the same axis: not which tool, but who may author it. An uncomfortable corollary follows either way: a harness component is worth having precisely because the executor has no way around it, and that is the same property which makes it model-coupled when a model is what wrote it.

The selection problem is worse than the transfer problem

Transfer is the headline. Selection is the finding that should actually change how you work.

HarnessDev ran nine evolution lineages producing 73 official harness versions and 64 adjacent version switches, then scored every switch twice: once on the feedback set the creator could see, and once on a held-out set of 630 SWE-bench Pro instances it never saw. Across those 64 switches, the visible score and the hidden score moved in the same direction only 34 times. That is close to a coin flip, and it means the signal the agent was optimizing against was roughly uninformative about whether the edit helped.

A different cut of the same 64 looks worse than the headline. Every official version had to complete both the SWE-bench Pro and the Terminal-Bench legs, so each switch can move on one benchmark, both, or neither:

• 8 regressed on both benchmarks
• 16 regressed on one
• 3 traded one benchmark against the other
• 7 produced no measurable change
• 27 reported gains that sit inside the run-to-run noise band of about plus or minus 4.75 points for the same commit
• 1 contained no executable code change
• 2 showed clear positive evidence beyond the noise

Two out of sixty-four. And when the agent picked its own final version, only 2 of 9 declared versions were the held-out optimal one.

Evolution did work, which is why the framing has to stay precise. All five self-runtime lineages improved on held-out tasks, by between 1.43 and 4.44 points, mean 3.11. Those gains are real and small. What failed is the ability to tell which edit produced them.

Two more numbers from the same audit, because they describe harnesses that many teams already have:

Components that never run. Of 108 component instances in the code domain, 72 fire in real runs, 18 have only partial evidence, and 18 are never observed at all. Every unobserved one concerns state and memory. In the wider artifact audit, 124 of 587 features in the writing domain are confirmed dead code, and of 169 new functions or classes added during evolution, 31 are reachable only through dead code and 25 have no caller at all.

The state layer is decorative. Eleven of eighteen artifacts define a State class. One exposes a state-saving interface. One implements periodic checkpointing. Across 26,679 recorded task trajectories, not a single checkpoint event appears. An agent asked to design durable state wrote the class and never wired it, which is the exact failure mode a human reviewer catches in ten seconds and a benchmark score never catches at all. The checkpointing patterns that make agent state survive a restart are not subtle, and the agent wrote none of them.

Three gates for anything the agent adds to the harness

The instrument for all of this shipped this month. claude plugin eval requires Claude Code v2.1.269 or later, and the two failure modes above each map onto a flag.

Run three gates on any harness component before it ships, whether an agent wrote it or you did. They sit below the level of testing whether your setup produces consistently good code: a gate here says one component earns its place, not that the system works.

Gate 1: does it ever fire? HarnessDev's 18 never-observed components are not hypothetical, and a component that never runs cannot be the reason anything improved. Say the component under test is a release-notes skill. A tool_used grader answers the question from the transcript at zero model cost. This is the grader file's frontmatter on its own, saved into the case's graders/ directory:

---type: tool_usedtool: Skillinput_match: '"skill"\s*:\s*"(?:[\w-]+:)?release-notes"'---

The docs are direct about what you usually learn: "The most common first finding is a Δ near zero with the case's tool_used: Skill grader failing, which means Claude isn't choosing your skill on natural phrasing." Running that check across a whole installed plugin is the subject of Superpowers Plugin for Claude Code: Install and Verify.

Gate 2: does it beat not having it, by more than noise? This is the 27-of-64 problem, and it is the default behaviour rather than something you configure. Each case runs three times with the component loaded and three times without, and you get WITH, W/OUT, and Δ. Anthropic's documentation states the consequence plainly: "If a case scores 1.0 both with and without the plugin, the plugin isn't what made it pass." Raise --runs when Δ is small enough that three runs cannot separate it from variance.

Gate 3: does its contribution survive an executor swap? Run Gate 2 under each model, which is one flag run twice:

claude plugin eval . --model claude-opus-5 --runs 5 --json opus.jsonclaude plugin eval . --model claude-haiku-4-5-20251001 --runs 5 --json haiku.json

Then compare the two deltas, not the two suite scores. Subtracting the suite scores is the obvious move and it is wrong, for the same reason this article raised against the paper two sections ago. Opus 5 and Haiku 4.5 sit far apart in raw capability, so that difference is mostly capability and the coupling never separates out. Gate 2 has already done the separating. Each run gives you a Δ whose no-component arm subtracts the capability effect, so what you compare is Δ under Opus against Δ under Haiku. A component worth +0.30 under one model and +0.02 under the other is coupled, whatever the two absolute scores were.

That comparison needs its own noise band, and a wider one than Gate 2's, because it compounds two independent run distributions. Take the spread across the five runs in each arm. A gap sitting inside it is not a finding, which is the 27-of-64 error committed one level up.

Two cautions. Pin both arms to dated model identifiers wherever your provider publishes them, because a floating alias moves the baseline underneath you between runs. And be precise about what this measures: a coupling delta between two executors, which is an Author Premium only when one of the arms is the model that authored the component. For a component you wrote yourself there is no author arm and no premium, just the coupling. Anthropic's documentation tells you to pin --model in continuous integration (CI) "so a model rollout isn't mistaken for a plugin regression," which is precisely the variable Gate 3 unpins on purpose.

Then apply the split:

This is the gate that the Skill Sharpening Loop was always missing. That loop turns operational failures into eval cases and eval cases into skill improvements, and it compounds precisely because a human decides what enters the loop. HarnessDev is the measurement showing what happens when nobody does: gains inside the noise band get promoted, dead components accumulate, and the executor the whole thing was fitted against becomes a dependency nobody wrote down.

There is a shipped precedent for narrowing it, too. Claude Code's bundled /verify skill writes what worked back into its own SKILL.md, and the documentation records that "before v2.1.205, the bundled skill told Claude to fold in anything a run learned, which caused frequent merge conflicts." The current version edits the file only when a run steered wrong. Merge conflicts are not the accumulation failure HarnessDev measures, and I will not claim they are. What the two share is the direction of the fix: a self-editing loop shipped with an open write condition, accumulated enough to become somebody's problem, and got a narrower one.

What would prove agent-written harnesses transfer

This thesis rests on a ten-day-old paper with a stated limitation that its evolution arm has one trajectory per cell and evaluates held-out performance on SWE-bench Pro only. They say outright that the trajectories "do not support uncertainty estimates or population-level comparisons." Four results would move it.

Start with the reverse swap. Nobody has run a Gemini-authored harness under Opus, and until someone does, "the harness is coupled to its author" is supported by one direction of one experiment. A null result in the other direction would mean the effect is partly about Gemini rather than about authorship.

Replicating the clean-sensitivity number at higher n would settle a second question. 33.0 and 49.1 differ by one degenerate run out of three. At five or ten runs per cell, one of those two numbers becomes the real one.

Harder, and more valuable, is a benchmark that separates the layers on purpose. Every current evolution benchmark either freezes the executable layer, as Lin et al. do, or evolves everything at once, as HarnessDev does. One experiment would settle the read/obey line: a single harness evolved twice against the same feedback, once advisory-only and once with binding logic unlocked, then both swapped. If the advisory-only arm holds its gains and the unlocked arm does not, the line is real and the ceiling for safe self-authorship has a number.

Nothing computes Gate 3 for you, which is why it is two commands and a subtraction. On the day claude plugin eval or an equivalent reports a cross-model delta as a first-class number, Author Premium stops being an argument and becomes a field in a report.

Until then the practical position is unchanged and it is not "stop letting agents write harness components." It is that your agent may write what it asks its executor to consider, and a human signs off on what it forces its executor to do.

References

• Wu, Y., Zhang, J., Shi, J., et al. (2026). HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness? arXiv:2609.01437v1 [cs.SE], 1 September 2026. https://arxiv.org/abs/2609.01437
• Lin, M., Wu, J., Wang, Z., et al. (2026). Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents. arXiv:2605.30621 [cs.AI], 28 May 2026. https://arxiv.org/abs/2605.30621
• Goyal, A., & Ray, J. (2026). Does Your Agent's Memory Survive a Model Upgrade? A Controlled Study of Memory Portability. arXiv:2609.05339 [cs.AI], 4 September 2026. https://arxiv.org/abs/2609.05339
• Wang, Y., Zhu, H., Hu, Z., et al. (2026). Rethinking the Evaluation of Harness Evolution for Agents. arXiv:2607.12227 [cs.AI], 14 July 2026. https://arxiv.org/abs/2607.12227
• Yao, Y., Tan, X., Liu, C.-H., et al. (2026). Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows. arXiv:2605.27922 [cs.AI], 27 May 2026. https://arxiv.org/abs/2605.27922
• Zheng, C., Xue, C., Liang, B., Yang, J., & Zhang, C. (2026). SEAGym: An Evaluation Environment for Self-Evolving LLM Agents. arXiv:2606.17546 [cs.AI], 16 June 2026. https://arxiv.org/abs/2606.17546
• Weng, L. (2026). Harness Engineering for Self-Improvement. Lil'Log, July 2026. https://lilianweng.github.io/posts/2026-07-04-harness/
• Anthropic. Test plugins with evals. Claude Code Documentation, retrieved 21 September 2026. https://code.claude.com/docs/en/plugin-evals
• Anthropic. Extend Claude with skills. Claude Code Documentation, retrieved 21 September 2026. https://code.claude.com/docs/en/skills
• Razzaq, A. (2026, September 11). Can LLMs Engineer Their Own Agent Harness? ByteDance Seed's HarnessDev Says Only 34 of 64 Changes Generalize. MarkTechPost. https://www.marktechpost.com/2026/09/11/can-llms-engineer-their-own-agent-harness-bytedance-seeds-harnessdev-says-only-34-of-64-changes-generalize/
• Deng, X., et al. (2025). SWE-bench Pro. arXiv:2509.16941. https://arxiv.org/abs/2509.16941
• Wei, J., et al. (2025). BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents. arXiv:2504.12516. https://arxiv.org/abs/2504.12516
• Chan, J. S., et al. (2024). MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering. arXiv:2410.07095. https://arxiv.org/abs/2410.07095
• The Terminal-Bench Team (2026). Terminal-Bench leaderboard. https://www.tbench.ai/ - a rolling leaderboard, showing Terminal-Bench 4.0 as of 21 September 2026; the 2.1 figures cited here are the ones HarnessDev evaluated against.

Related Articles

Agentic AI

• Superpowers Plugin for Claude Code: Install and Verify
• Which Claude Code Layer Solves Your Problem? A Diagnostic Guide for AI Engineers
• Agent Skills Are Not Prompts. They Are Production Knowledge Infrastructure.
• Skills vs Hooks in Claude Code: Enforceability Is the Design Variable