Stop Asking Which Coding Model Is Best

Most coding-agent discussions still start with the wrong question:

Which model is best?

That question is not useless. A weak model inside a great workflow is still a weak model.

But it is no longer the whole game.

The sharper question is:

What is the harness allowed to change?

Because the model is only one part of the agent run. The harness decides what context gets loaded, which tools exist, when tests run, how failures are summarized, what gets retried, when a human is asked, which model handles which subtask, and what state survives into the next run.

That is where the current research signal is getting interesting.

TTHE: Test-Time Harness Evolution treats the executable harness itself as mutable test-time state. SkillOpt treats skill files as trainable external state for frozen agents. The Harness Effect argues that orchestration design can move token cost more than model choice on a controlled workload. Google’s Android Bench move to Harbor is another version of the same idea: the evaluation plumbing around the model changes what “good” even means.

Different artifacts. Same direction.

The leverage layer is moving outside the model.

1

model stays frozen

N

harness variants compete

0

trust in unscored self-improvement

The harness is the part that touches reality

A coding model predicts text.

A coding agent changes a repo.

The difference is the harness.

The harness controls the run

• Context: which files, traces, specs, issues, memories, and prior attempts enter the prompt.
• Tools: which commands, editors, browsers, APIs, and sandboxes the model can use.
• Verification: which tests, type checks, linters, probes, review gates, and diff checks decide progress.
• Recovery: what happens after failure, timeout, flaky tests, tool errors, or unclear requirements.
• Persistence: which lessons, skills, traces, and policies survive into future tasks.

That is why “just use the better model” is an incomplete operating strategy.

Two teams can run the same model and get different outcomes because one harness gives the model the right file slice, forces the right tests, blocks unsafe commands, compacts traces cleanly, and routes cheap subproblems away from expensive calls.

The other harness just keeps pasting more context into the next turn.

What TTHE changes

TTHE is interesting because it moves beyond “we optimized a prompt before deployment.”

Its claim is more dynamic:

failed or weak runs
  -> execution traces
  -> proposed harness edits
  -> proxy-judged selection
  -> persisted better harness
  -> future tasks use the new wrapper

The model weights do not change. The task answers are not used as labels. The adaptation happens in the executable wrapper around the model.

That wrapper can change how context is assembled, how tools are sequenced, how intermediate outputs are verified, or how the agent recovers from failure.

Harness evolution loop

the selected wrapper, not just the answer, survives

1. 01
   Run tasks
2. 02
   Collect traces
3. 03
   Propose bounded harness edits
4. 04
   Judge with proxy signals
5. 05
   Promote or reject
6. 06
   Persist next harness

This is the piece normal teams should steal, carefully.

Not “let the agent rewrite itself in production.”

More like:

Our coding agent keeps failing in the same way.
Which small harness change would reduce that failure next week?

Maybe the answer is not a better model. Maybe it is:

• always load the failing test before editing
• summarize shell errors into a structured failure note
• run a cheaper search model before the expensive patch model
• block broad refactors unless the issue explicitly asks for them
• require a rollback plan before touching migrations
• persist a skill when three traces show the same repair pattern

That is a product surface.

Skills are harness state too

SkillOpt points at a softer version of the same pattern.

Instead of evolving executable control code, it optimizes a natural-language skill document. The target agent stays frozen. A separate optimizer proposes bounded edits. A held-out validation gate decides whether the skill changed for the better. The deployed artifact is a compact best_skill.md, not a new model.

That matters because many teams already have this artifact, even if they do not call it a skill.

They have:

• AGENTS.md
• CLAUDE.md
• repo playbooks
• review checklists
• tool policies
• debugging recipes
• internal “how we work” docs

Right now those files are often edited by taste.

SkillOpt’s useful frame is: treat those documents as trainable external state. Score them. Version them. Reject bad edits. Promote only the ones that survive held-out tasks.

Prompt file vs. trainable artifact

Manual prompt file

• Edited when someone has a hunch.
• Accepted because it sounds reasonable.
• Grows until nobody reads it.
• Hard to compare across versions.

Measured skill artifact

• Edited because traces show a repeated failure.
• Accepted because validation improves.
• Bounded by size, scope, and task fit.
• Versioned against a frozen eval set.

Cost is a harness problem

The Harness Effect paper is useful even if you are skeptical of the exact savings generalizing everywhere.

The important claim is not the percentage. The important claim is the mechanism.

Agent cost is not only:

model price * tokens

It is also:

how much context the harness repeats
how often it retries blindly
how it caches stable instructions
how it stores state outside the prompt
which tasks get routed to which model
how quickly it stops doomed runs

The paper reports that changing only the orchestration layer reduced blended cost, latency, and token use on its workload while holding models constant. Treat that as a strong hint, not a universal law.

For builders, the lesson is immediate: if your agent bill is ugly, do not only shop for cheaper tokens. Inspect the harness.

Where to look before switching models

Use when

• ✓ Measure tokens per successful task, not tokens per call.
• ✓ Separate stable instructions from volatile task context so caching can work.
• ✓ Store bulky evidence in files or state, then retrieve only the needed slice.
• ✓ Add failure-spend limits so bad loops stop early.
• ✓ Route cheap classification, search, and summarization away from premium coding calls.

Avoid when

• × Treat a longer context window as a cost strategy.
• × Retry the same failing loop with slightly louder instructions.
• × Compare models without holding the harness constant.
• × Let model-judged success hide missing tests or unsafe traces.
• × Optimize for leaderboard score while ignoring dollars per accepted patch.

The eval harness is part of the product

Android Bench moving to Harbor is not the same kind of artifact as TTHE or SkillOpt, but it belongs in the same story.

If your benchmark uses one agent loop, one verifier, one task format, and one scoring policy, then it is measuring that whole setup. Change the framework and you may change the result.

That is not a scandal. That is the point.

For agents, evaluation is not a neutral afterthought. It is part of the system definition.

Android Bench’s public repository describes a two-stage setup: inference generates a patch, then a verifier applies the patch and runs tests in a standardized environment. That benchmark plumbing is doing real work. It decides what counts as solving an Android development task.

So when someone asks “which coding model is best?”, the honest answer is:

Best under which harness, verifier, task set, tool policy, and cost budget?

A practical harness review

If you are building agents, do this before you swap models again.

Pick one repeated workflow. Bug fix. Dependency update. Support triage. Data cleanup. Test repair.

Then write down the harness as if it were production code.

context inputs:
tools:
permissions:
verification gates:
retry policy:
escalation rule:
cost budget:
persistent artifacts:
rollback path:

Now ask which parts are allowed to change.

Good harness changes are bounded

• Change context assembly, not the whole product architecture.
• Change tool order, not the safety boundary.
• Change verification prompts, not the hard verifier.
• Change retry policy, not the definition of success.
• Change skill text, not production permissions.

The key is separation.

Let the adaptive layer improve the boring operational details. Keep the authority layer boring on purpose.

The harness may learn that it should inspect failing tests before editing.

It should not learn that it may bypass review.

The harness may learn that a cheaper model is enough for issue classification.

It should not learn that production secrets are useful context.

The harness may learn a better recovery pattern for flaky commands.

It should not learn a new definition of “done” without a human.

The new question

Model choice still matters.

But if two models are close, the harness can dominate the actual product experience:

• fewer wasted tokens
• fewer fake completions
• better trace evidence
• safer tool use
• faster recovery from repeated failures
• durable improvements that survive the next task

That is why the question should change.

Not:

Which model won the leaderboard?

Ask:

What harness was it running under?
What did the verifier measure?
What external state was allowed to improve?
What can be rolled back?
What gets cheaper as the system learns?

That is the engineering layer where agent products will differentiate.

The model is still the engine.

But the harness is becoming the transmission, the dashboard, the brakes, the maintenance log, and increasingly the part that learns how the system should drive.