# Stop Asking Which Coding Model Is Best Canonical URL: https://huecki.com/en/blog/stop-asking-which-coding-model-is-best/ Markdown URL: https://huecki.com/en/blog/stop-asking-which-coding-model-is-best.md Language: English Published: 2026-07-10 Updated: 2026-07-10 Author: Dominic Hückmann Topic: AI Agent Workflows - Agent topics: Agent Harnesses, Context Engineering, Agent Evals - Tags: AI Agents, Agent Harness, Coding Agents, Agent Evals, AI Engineering - Audience: developers building coding agents, AI product engineers, teams evaluating agentic workflows - Concepts: agent-harnesses, coding-agent-evals, external-agent-state, test-time-adaptation Thesis: The next practical advantage in coding agents is not only model choice. It is the harness that assembles context, controls tools, verifies work, learns from traces, and decides what persists. Content status: field-note ## Summary The useful question is moving from which model is best to what your agent harness can change, measure, persist, and roll back. ## Description TTHE, SkillOpt, the Harness Effect, and Android Bench all point to the same practical shift: the model is no longer the only thing to optimize. The harness around the model is becoming the leverage layer. ## Body Most coding-agent discussions still start with the wrong question: ```txt Which model is best? ``` That question is not useless. A weak model inside a great workflow is still a weak model. But it is no longer the whole game. The sharper question is: ```txt What is the harness allowed to change? ``` Because the model is only one part of the agent run. The harness decides what context gets loaded, which tools exist, when tests run, how failures are summarized, what gets retried, when a human is asked, which model handles which subtask, and what state survives into the next run. That is where the current research signal is getting interesting. [TTHE: Test-Time Harness Evolution](https://arxiv.org/abs/2607.08124) treats the executable harness itself as mutable test-time state. [SkillOpt](https://arxiv.org/abs/2605.23904) treats skill files as trainable external state for frozen agents. [The Harness Effect](https://arxiv.org/abs/2607.06906) argues that orchestration design can move token cost more than model choice on a controlled workload. Google's [Android Bench move to Harbor](https://android-developers.googleblog.com/2026/07/android-bench-llm-measurement.html) is another version of the same idea: the evaluation plumbing around the model changes what "good" even means. Different artifacts. Same direction. The leverage layer is moving outside the model. ## The harness is the part that touches reality A coding model predicts text. A coding agent changes a repo. The difference is the harness. That is why "just use the better model" is an incomplete operating strategy. Two teams can run the same model and get different outcomes because one harness gives the model the right file slice, forces the right tests, blocks unsafe commands, compacts traces cleanly, and routes cheap subproblems away from expensive calls. The other harness just keeps pasting more context into the next turn. ## What TTHE changes TTHE is interesting because it moves beyond "we optimized a prompt before deployment." Its claim is more dynamic: ```txt failed or weak runs -> execution traces -> proposed harness edits -> proxy-judged selection -> persisted better harness -> future tasks use the new wrapper ``` The model weights do not change. The task answers are not used as labels. The adaptation happens in the executable wrapper around the model. That wrapper can change how context is assembled, how tools are sequenced, how intermediate outputs are verified, or how the agent recovers from failure. This is the piece normal teams should steal, carefully. Not "let the agent rewrite itself in production." More like: ```txt Our coding agent keeps failing in the same way. Which small harness change would reduce that failure next week? ``` Maybe the answer is not a better model. Maybe it is: - always load the failing test before editing - summarize shell errors into a structured failure note - run a cheaper search model before the expensive patch model - block broad refactors unless the issue explicitly asks for them - require a rollback plan before touching migrations - persist a skill when three traces show the same repair pattern That is a product surface. ## Skills are harness state too SkillOpt points at a softer version of the same pattern. Instead of evolving executable control code, it optimizes a natural-language skill document. The target agent stays frozen. A separate optimizer proposes bounded edits. A held-out validation gate decides whether the skill changed for the better. The deployed artifact is a compact `best_skill.md`, not a new model. That matters because many teams already have this artifact, even if they do not call it a skill. They have: - `AGENTS.md` - `CLAUDE.md` - repo playbooks - review checklists - tool policies - debugging recipes - internal "how we work" docs Right now those files are often edited by taste. SkillOpt's useful frame is: treat those documents as trainable external state. Score them. Version them. Reject bad edits. Promote only the ones that survive held-out tasks. ## Cost is a harness problem The Harness Effect paper is useful even if you are skeptical of the exact savings generalizing everywhere. The important claim is not the percentage. The important claim is the mechanism. Agent cost is not only: ```txt model price * tokens ``` It is also: ```txt how much context the harness repeats how often it retries blindly how it caches stable instructions how it stores state outside the prompt which tasks get routed to which model how quickly it stops doomed runs ``` The paper reports that changing only the orchestration layer reduced blended cost, latency, and token use on its workload while holding models constant. Treat that as a strong hint, not a universal law. For builders, the lesson is immediate: if your agent bill is ugly, do not only shop for cheaper tokens. Inspect the harness. ## The eval harness is part of the product Android Bench moving to Harbor is not the same kind of artifact as TTHE or SkillOpt, but it belongs in the same story. If your benchmark uses one agent loop, one verifier, one task format, and one scoring policy, then it is measuring that whole setup. Change the framework and you may change the result. That is not a scandal. That is the point. For agents, evaluation is not a neutral afterthought. It is part of the system definition. Android Bench's public repository describes a two-stage setup: inference generates a patch, then a verifier applies the patch and runs tests in a standardized environment. That benchmark plumbing is doing real work. It decides what counts as solving an Android development task. So when someone asks "which coding model is best?", the honest answer is: ```txt Best under which harness, verifier, task set, tool policy, and cost budget? ``` ## A practical harness review If you are building agents, do this before you swap models again. Pick one repeated workflow. Bug fix. Dependency update. Support triage. Data cleanup. Test repair. Then write down the harness as if it were production code. ```txt context inputs: tools: permissions: verification gates: retry policy: escalation rule: cost budget: persistent artifacts: rollback path: ``` Now ask which parts are allowed to change. The key is separation. Let the adaptive layer improve the boring operational details. Keep the authority layer boring on purpose. The harness may learn that it should inspect failing tests before editing. It should not learn that it may bypass review. The harness may learn that a cheaper model is enough for issue classification. It should not learn that production secrets are useful context. The harness may learn a better recovery pattern for flaky commands. It should not learn a new definition of "done" without a human. ## The new question Model choice still matters. But if two models are close, the harness can dominate the actual product experience: - fewer wasted tokens - fewer fake completions - better trace evidence - safer tool use - faster recovery from repeated failures - durable improvements that survive the next task That is why the question should change. Not: ```txt Which model won the leaderboard? ``` Ask: ```txt What harness was it running under? What did the verifier measure? What external state was allowed to improve? What can be rolled back? What gets cheaper as the system learns? ``` That is the engineering layer where agent products will differentiate. The model is still the engine. But the harness is becoming the transmission, the dashboard, the brakes, the maintenance log, and increasingly the part that learns how the system should drive. ## FAQ ### What is an agent harness? An agent harness is the executable system around a model: context assembly, tools, permissions, memory, verification, retries, traces, and recovery logic. ### What does it mean for a harness to evolve? It means the wrapper around the model changes in controlled ways, such as tool order, context strategy, verification rules, retry policy, or recovery logic, while the model weights stay fixed. ### What is the main risk? Bad proxy signals. If the judge rewards the wrong trace pattern, the harness can get better at passing the proxy while getting worse at the real task. ## Related - Earlier framing of the harness as the runtime around the model.: agents-dont-need-longer-prompts-they-need-harnesses - How to evaluate skills, traces, tools, and the whole agent run.: measure-agentic-setups-skills - Why external evaluation systems matter more than self-improvement vibes.: agent-is-not-the-product-fitness-function-is ## Source References - [TTHE: Test-Time Harness Evolution](https://arxiv.org/abs/2607.08124) (paper) - [SkillOpt: Executive Strategy for Self-Evolving Agent Skills](https://arxiv.org/abs/2605.23904) (paper) - [microsoft/SkillOpt](https://github.com/microsoft/SkillOpt) (repository) - [SkillOpt: Agent skills as trainable parameters](https://www.microsoft.com/en-us/research/blog/skillopt-agent-skills-as-trainable-parameters/) (official) - [The Harness Effect: How Orchestration Design Sets the Token Economics of Enterprise Agentic AI](https://arxiv.org/abs/2607.06906) (paper) - [Evolving how LLMs are measured for Android: the next era of Android Bench](https://android-developers.googleblog.com/2026/07/android-bench-llm-measurement.html) (official) - [android-bench/android-bench](https://github.com/android-bench/android-bench) (repository)