# Single-Turn Evals Don’t Teach Your Agent Enough Canonical URL: https://huecki.com/en/blog/single-turn-evals-dont-teach-agent-enough/ Markdown URL: https://huecki.com/en/blog/single-turn-evals-dont-teach-agent-enough.md Language: English Published: 2026-08-23 Updated: 2026-08-23 Author: Dominic Hückmann Topic: AI Agent Workflows - Agent topics: Context Engineering, Agent Evals - Tags: AI Agents, Agent Skills, Agent Evals, AI Engineering, Automation - Audience: agent-harness builders, support automation teams, developers maintaining agent skills, AI product engineers - Concepts: SkillEvo, multi-turn evals, agent skills, skill governance, failure attribution Thesis: A failed agent conversation should become a governed skill patch, not another one-shot QA eval. Content status: field-note ## Summary Single-turn evals expose the first obvious skill gap, then stop teaching the system. Use multi-turn failure replay, repairable attribution, bounded edits, and a separate governance check instead. ## Description A practical field note from SkillEvo: if you want agent skills to improve, replay failures as multi-turn conversations, attribute only repairable gaps, and govern every patch. ## Body Most agent-skill evals are too polite. They ask one question. The agent answers. A judge gives a score. If the score is bad, the skill gets patched. That catches the obvious gap. It misses the dangerous one: the gap that only appears after the agent gives a plausible answer, the user follows that answer, and the conversation walks into a second-order failure. [SkillEvo](https://arxiv.org/html/2608.13120v1), a new paper from Tencent Cloud Andon and Zhejiang University, is useful because it names that problem clearly. The authors argue that skill evolution stalls when feedback comes from single-turn question answering. Once the first visible gap is patched, the feedback signal stops supplying new information. Multi-turn interaction changes the shape of the test. A follow-up question can expose a defect that the first question could not reach. That is the workflow worth stealing. ## The short version If an agent skill fails in conversation, do not patch it from one QA pair. Replay the failure as a multi-turn scenario. Then ask four questions: 1. Did the simulated user actually raise every important intent? 2. Did the agent answer the raised intents correctly? 3. Is the failure repairable by the skill, or was it a tool, permission, product, or evaluation problem? 4. Did the patch preserve the old stable facts and the structure of the skill? Only then should the skill change. ## Why single-turn feedback decays A one-shot eval is good at finding the first missing fact. Example: ```txt User: When I renew this cloud package, does the new quota apply immediately? Agent: Yes. ``` If that answer is wrong, a single-turn eval can catch it. But many support failures do not stop there. The user asks a follow-up. The agent builds on its own wrong premise. The failure becomes more specific, more operational, and more damaging. SkillEvo's case study shows exactly this pattern: the first answer about a renewed traffic package is plausibly self-consistent but wrong. The simulated user then asks what happens if the remaining traffic runs out before the reset date. That second turn exposes the real operational harm. This is why the paper's useful claim is not "multi-turn is more realistic." The useful claim is sharper: **Each repaired gap lets the next conversation reach a deeper gap.** Single-turn evals see the visible surface. Multi-turn replay keeps reopening the failure boundary. ## The maintenance loop to steal SkillEvo's full loop is more elaborate than most small teams need, but its control points translate well. The attribution gate matters most. A failed conversation can come from different causes: - The skill missed a stable product rule. - The agent lacked permission to perform an action. - The tool failed. - The simulator skipped an intent. - The judge produced a false negative. Only the first category should become skill content. If you encode tool limits, simulator mistakes, or one weird evaluation miss into the skill, you do not get learning. You get a messy knowledge file with false confidence. ## A small-team version You do not need Tencent's production setup to copy the shape. Start with one recurring support, ops, or coding-agent workflow where you already have failure traces. ```txt For each failed conversation: 1. Extract the user goal, facts already known, and expected human resolution. 2. Replay the case as a bounded multi-turn scenario. 3. Track which key intents were raised and answered. 4. Label each failure: - Knowledge Gap - Capability Limit - Evaluation Noise 5. Patch only Knowledge Gap cases. 6. Anchor the patch to the previous accepted skill. 7. Run a separate governance check before promotion. ``` The boring part is the product. The maintenance loop is useful because it refuses to treat every failure as missing knowledge. ## Govern the patch, not just the score This is the second good idea in SkillEvo. Most teams use a scalar gate: ```txt new skill score >= old skill score ``` That is not enough. A score can tell you the candidate got better on the eval. It cannot tell you that a stable old fact was deleted, a reference now points nowhere, or a precise rule was generalized into a vague sentence. SkillEvo treats a skill as a structured knowledge system. Its governor checks for three boring but real failure modes: - **Knowledge bloat:** repeated patches add redundant text and dilute routing. - **Reference breakage:** files, sections, or internal links stop lining up. - **Factual over-generalization:** concrete rules decay into vague advice. For small teams, the simplest version is a pull-request-style review for skills: ```txt Patch scope: - Which failure trace caused this edit? - Which exact knowledge gap is repaired? - Which old facts must remain true? - Which references changed? - What replay proves the fix? - What regression replay proves old behavior still works? ``` No clean answer, no promotion. ## Where not to copy the paper blindly The paper reports strong results: across six cloud-service categories, 9 production Skills, and 98 reference files, SkillEvo improves over the original skills by 51.8 points, over self-reflection by 23.0 points, and over single-turn-QA evolution by 15.4 points. Treat those as author-reported results, not a universal constant. The dataset comes from production support tickets and cannot be released. The domain is Tencent Cloud service support, not every agent system. The paper gives implementation details, prompts, hyperparameters, and a case study, but you still cannot reproduce the exact benchmark without the private ticket set. That does not weaken the workflow lesson. It just narrows the claim. ## The practical takeaway A failed agent conversation is not just a bad answer. It is a possible maintenance signal. But the signal has to be cleaned before it becomes memory: ```txt multi-turn replay -> intent coverage -> answer accuracy -> repairable attribution -> bounded patch -> governance check -> human rollout ``` That is the difference between a skill bank and a pile of model-written notes. Single-turn evals can tell you where the first visible crack is. Multi-turn replay tells you whether the repair holds when the user keeps going. ## Sources - [SkillEvo: Self-Renewing Evolution Gradients from Multi-Turn Interaction Feedback](https://arxiv.org/html/2608.13120v1) - [arXiv abstract page](https://arxiv.org/abs/2608.13120) - [Hugging Face paper page](https://huggingface.co/papers/2608.13120) - Research artifact: `content-research/deepresearch/skillevo-multi-turn-skill-evolution-2026-08-23/report.md` ## FAQ ### What is the main idea behind SkillEvo? SkillEvo turns failed multi-turn support interactions into feedback for improving agent skills, then uses a separate governance layer to prevent factual and structural degradation. ### Why are single-turn evals not enough for agent skills? A single question often exposes only the first visible gap. Follow-up turns can reveal hidden defects that appear only after the agent gives a plausible but incomplete or wrong answer. ### What should teams copy from the paper? Copy the maintenance loop: replay failures as multi-turn scenarios, check intent coverage, attribute only repairable knowledge gaps, edit inside evidence boundaries, and require governance plus human confirmation before rollout. ## Related - FlowEvo field note on turning successful traces into reusable skills.: stop-losing-agent-workflows - Broader guide to evaluating skills, tools, traces, and agent systems.: measure-agentic-setups-skills - Release-process frame for shipping prompt, tool, memory, and harness changes.: your-agent-harness-needs-a-release-process ## Source References - [SkillEvo: Self-Renewing Evolution Gradients from Multi-Turn Interaction Feedback](https://arxiv.org/html/2608.13120v1) (paper) - [SkillEvo arXiv abstract](https://arxiv.org/abs/2608.13120) (paper) - [Hugging Face paper page: SkillEvo](https://huggingface.co/papers/2608.13120) (paper-index)