Single-Turn Evals Don’t Teach Your Agent Enough

Most agent-skill evals are too polite.

They ask one question. The agent answers. A judge gives a score. If the score is bad, the skill gets patched.

That catches the obvious gap.

It misses the dangerous one: the gap that only appears after the agent gives a plausible answer, the user follows that answer, and the conversation walks into a second-order failure.

SkillEvo, a new paper from Tencent Cloud Andon and Zhejiang University, is useful because it names that problem clearly. The authors argue that skill evolution stalls when feedback comes from single-turn question answering. Once the first visible gap is patched, the feedback signal stops supplying new information.

Multi-turn interaction changes the shape of the test. A follow-up question can expose a defect that the first question could not reach.

That is the workflow worth stealing.

9

production Skills in the paper

98

skill-reference files evaluated

+15.4

author-reported points over single-turn QA evolution

The short version

If an agent skill fails in conversation, do not patch it from one QA pair.

Replay the failure as a multi-turn scenario.

Then ask four questions:

1. Did the simulated user actually raise every important intent?
2. Did the agent answer the raised intents correctly?
3. Is the failure repairable by the skill, or was it a tool, permission, product, or evaluation problem?
4. Did the patch preserve the old stable facts and the structure of the skill?

Only then should the skill change.

Failed conversation to governed skill patch

evaluation should feed repair, not just scoring

1. 01
   Real failed ticket
2. 02
   Multi-turn replay
3. 03
   Intent coverage
4. 04
   Repairable attribution
5. 05
   Bounded skill edit
6. 06
   Governance check
7. 07
   Human rollout

Why single-turn feedback decays

A one-shot eval is good at finding the first missing fact.

Example:

User: When I renew this cloud package, does the new quota apply immediately?
Agent: Yes.

If that answer is wrong, a single-turn eval can catch it.

But many support failures do not stop there. The user asks a follow-up. The agent builds on its own wrong premise. The failure becomes more specific, more operational, and more damaging.

SkillEvo’s case study shows exactly this pattern: the first answer about a renewed traffic package is plausibly self-consistent but wrong. The simulated user then asks what happens if the remaining traffic runs out before the reset date. That second turn exposes the real operational harm.

This is why the paper’s useful claim is not “multi-turn is more realistic.” The useful claim is sharper:

Each repaired gap lets the next conversation reach a deeper gap.

Single-turn evals see the visible surface. Multi-turn replay keeps reopening the failure boundary.

The maintenance loop to steal

SkillEvo’s full loop is more elaborate than most small teams need, but its control points translate well.

Four gates for skill repair

• Coverage: the replay must prove that key user intents were actually raised.
• Accuracy: judge the agent only on intents that were exposed in the conversation.
• Attribution: patch only failures caused by a repairable knowledge gap.
• Governance: reject or repair edits that delete stable facts, break references, or bloat the skill.

The attribution gate matters most.

A failed conversation can come from different causes:

• The skill missed a stable product rule.
• The agent lacked permission to perform an action.
• The tool failed.
• The simulator skipped an intent.
• The judge produced a false negative.

Only the first category should become skill content.

If you encode tool limits, simulator mistakes, or one weird evaluation miss into the skill, you do not get learning. You get a messy knowledge file with false confidence.

A small-team version

You do not need Tencent’s production setup to copy the shape.

Start with one recurring support, ops, or coding-agent workflow where you already have failure traces.

For each failed conversation:
1. Extract the user goal, facts already known, and expected human resolution.
2. Replay the case as a bounded multi-turn scenario.
3. Track which key intents were raised and answered.
4. Label each failure:
   - Knowledge Gap
   - Capability Limit
   - Evaluation Noise
5. Patch only Knowledge Gap cases.
6. Anchor the patch to the previous accepted skill.
7. Run a separate governance check before promotion.

The boring part is the product.

The maintenance loop is useful because it refuses to treat every failure as missing knowledge.

Bad skill evolution vs. governed skill evolution

One-shot patching

• One QA pair drives the edit.
• Every miss becomes more text.
• A scalar score accepts or rejects the whole candidate.
• The skill grows until nobody trusts it.

Governed replay

• A multi-turn failure trace drives the edit.
• Only repairable knowledge gaps become skill changes.
• Separate checks localize coverage, accuracy, attribution, and structure.
• A governor checks bloat, broken references, and factual drift.

Govern the patch, not just the score

This is the second good idea in SkillEvo.

Most teams use a scalar gate:

new skill score >= old skill score

That is not enough.

A score can tell you the candidate got better on the eval. It cannot tell you that a stable old fact was deleted, a reference now points nowhere, or a precise rule was generalized into a vague sentence.

SkillEvo treats a skill as a structured knowledge system. Its governor checks for three boring but real failure modes:

• Knowledge bloat: repeated patches add redundant text and dilute routing.
• Reference breakage: files, sections, or internal links stop lining up.
• Factual over-generalization: concrete rules decay into vague advice.

For small teams, the simplest version is a pull-request-style review for skills:

Patch scope:
- Which failure trace caused this edit?
- Which exact knowledge gap is repaired?
- Which old facts must remain true?
- Which references changed?
- What replay proves the fix?
- What regression replay proves old behavior still works?

No clean answer, no promotion.

Where not to copy the paper blindly

The paper reports strong results: across six cloud-service categories, 9 production Skills, and 98 reference files, SkillEvo improves over the original skills by 51.8 points, over self-reflection by 23.0 points, and over single-turn-QA evolution by 15.4 points.

Treat those as author-reported results, not a universal constant.

The dataset comes from production support tickets and cannot be released. The domain is Tencent Cloud service support, not every agent system. The paper gives implementation details, prompts, hyperparameters, and a case study, but you still cannot reproduce the exact benchmark without the private ticket set.

That does not weaken the workflow lesson. It just narrows the claim.

Use SkillEvo as a maintenance pattern

Do

• ✓ Use multi-turn replay when the original task is multi-turn.
• ✓ Separate simulator failure from agent failure.
• ✓ Patch only repairable knowledge gaps.
• ✓ Keep a separate governor outside the editor model.
• ✓ Require human confirmation before production rollout.

Do not

• × Treat one failed answer as enough evidence for a durable skill edit.
• × Encode permission limits or tool outages as knowledge.
• × Let the same model freely write and approve its own skill changes.
• × Trust a higher score if the skill lost old facts or broke references.
• × Automerge self-evolved skills into user-facing systems.

The practical takeaway

A failed agent conversation is not just a bad answer.

It is a possible maintenance signal.

But the signal has to be cleaned before it becomes memory:

multi-turn replay
-> intent coverage
-> answer accuracy
-> repairable attribution
-> bounded patch
-> governance check
-> human rollout

That is the difference between a skill bank and a pile of model-written notes.

Single-turn evals can tell you where the first visible crack is.

Multi-turn replay tells you whether the repair holds when the user keeps going.

Sources

• SkillEvo: Self-Renewing Evolution Gradients from Multi-Turn Interaction Feedback
• arXiv abstract page
• Hugging Face paper page
• Research artifact: content-research/deepresearch/skillevo-multi-turn-skill-evolution-2026-08-23/report.md