Your Agent Needs a Wiki Before It Writes Better Skills

The ELI5 version

Imagine a junior developer keeps a notebook while learning a codebase.

On Monday, they break the build because they edited a generated file.

On Tuesday, they break it again because they forgot the migration command.

On Wednesday, they finally write a clear checklist:

For API client changes:
1. Edit the schema, not the generated client.
2. Run the generator.
3. Run the API contract tests.
4. Check the generated diff.

That checklist is useful because it is not just a random note from one bad day. It is a repeated lesson, cleaned up into a reusable procedure.

That is the practical idea behind WikiSkill.

Do not let an agent jump straight from messy experience to permanent behavior.

First, keep the raw experience. Then compile the repeated lessons into a wiki. Then update the skill.

Why this matters

Agent skills are useful because they package practical know-how: instructions, workflows, scripts, examples, and “when to use this” rules.

The hard part is keeping those skills good.

If a human writes every skill by hand, the system learns slowly. If the agent rewrites its own skills after every run, the system can learn nonsense quickly.

WikiSkill is interesting because it inserts a middle layer.

Raw runs to better skills

the wiki is the memory that survives between skill edits

1. 01
   Raw traces
2. 02
   Wiki patterns
3. 03
   Skill proposal
4. 04
   Validation gate
5. 05
   Accept or roll back

The paper separates the workspace into three layers:

• Raw layer: immutable execution traces from agent runs.
• Wiki layer: persistent pattern pages, logs, and skill-impact history.
• Skills layer: executable procedural instructions that future agents use.

That separation is the part developers should steal.

Raw traces answer: what happened?

The wiki answers: what have we learned across runs?

Skills answer: what should the next agent do differently?

The trap: one run is not wisdom

After a failed run, an agent can always invent a lesson.

Sometimes the lesson is right:

Do not edit generated files directly.

Sometimes it is too broad:

Never touch generated files.

Sometimes it is actively harmful:

If the build fails, skip generated files in tests.

This is why a wiki layer matters. It gives the system somewhere to collect repeated evidence before changing the instructions future agents inherit.

Without vs. with a wiki layer

Direct skill edits

• A single failed run becomes a rule.
• Rejected edits are forgotten.
• The skill file becomes a messy log.
• Rollback loses the lesson too.

Wiki-informed skills

• Several traces become a scoped pattern.
• Skill-impact history records what failed before.
• The wiki holds evidence; the skill stays concise.
• Bad skill changes roll back, but the wiki keeps the evidence.

The last point is subtle and important.

In WikiSkill, candidate skill changes are gated by validation. If a proposed skill hurts performance, the skill can be rolled back. But the wiki persists. The system still remembers that the attempted fix failed, so it does not keep proposing the same bad edit.

That is closer to how good engineering teams learn.

You can revert a bad code change without deleting the incident write-up.

What the paper found

WikiSkill evaluates this idea across five benchmarks: mathematical reasoning, web search, spreadsheet manipulation, long-context document question answering, and interactive embodied tasks.

The authors report that WikiSkill improves over no-skill baselines in most model-benchmark settings and outperforms prior skill-evolution methods in their experiments.

Two findings are especially useful for builders:

3 layers

raw traces, wiki knowledge, active skills

5

benchmark families in the paper

12.3-23.9%

reported average gains within Qwen model scale

First, skills and model scale are complementary. The paper reports larger gains for stronger Qwen models: 12.3%, 17.5%, and 23.9% average improvement for 4B, 9B, and 27B models respectively.

Second, skill discovery and skill execution are not the same ability. In one reported ALFWorld result, a 9B model did better with a skill evolved by a 27B model than with its own self-evolved skill.

Translated into engineering terms:

Use strong agents to discover workflows.
Let cheaper agents execute the workflows when they can.

That is not guaranteed for every task. But it is a good architecture hint.

A practical version for your team

You do not need the full research setup to copy the useful pattern.

Start with three folders in your agent workspace:

agent-learning/
  raw/
    2026-08-29-run-184.md
  wiki/
    patterns/
      generated-files.md
      migration-order.md
    skill-impact.md
  skills/
    api-client-change/
      SKILL.md
      PURPOSE.md

Keep raw traces append-only. They are evidence, not polished truth.

Use wiki pattern pages for repeated lessons:

# Generated Client Changes

Observed in:
- run-184: direct edit disappeared after generator ran
- run-211: tests passed locally but CI regenerated client

Pattern:
Generated clients are outputs. Durable changes must start from the schema.

Useful procedure:
1. Edit `schemas/order-api.yaml`.
2. Run `npm run generate:order-client`.
3. Run `npm test -- order-client`.
4. Inspect both schema and generated diff.

Known exceptions:
- Fixture snapshots under `test/generated-fixtures/` are manually reviewed.

Then make the skill short:

# API Client Change Skill

Use when changing generated API clients.

Procedure:
1. Find the source schema before editing generated files.
2. Change the schema or generator input.
3. Run the generator.
4. Run contract tests.
5. Report schema diff, generated diff, and test command.

The wiki can be verbose. The skill should be compact.

The release gate

Do not promote every suggested skill edit.

Use a small validation set. For a coding agent, that can be five saved tasks:

• generated client change;
• migration change;
• UI route update;
• failing test fix;
• dependency upgrade.

When the agent proposes a skill update, run those tasks or replay fixtures. Accept the update only if it improves the target case and does not regress the others.

Minimum useful gate

• Every skill change links back to one or more wiki patterns.
• Every wiki pattern links back to raw traces or concrete artifacts.
• Every accepted skill change records before and after validation scores.
• Every rejected skill change stays in skill-impact history so it is not retried blindly.

This is the part many self-improving agent demos skip. The agent should be allowed to propose learning. Promotion is a separate operation.

Where this can go wrong

A wiki can become junk.

If it stores every thought, it becomes another messy chat log. If it stores untrusted content without provenance, it can preserve prompt injections or stale assumptions. If it keeps only successes, it loses the negative evidence that prevents repeated bad edits.

Keep the wiki useful

Do

• ✓ Separate raw traces, wiki patterns, and executable skills.
• ✓ Keep raw traces immutable and source-backed.
• ✓ Record rejected skill edits and why they failed.
• ✓ Keep skills short enough for agents to actually follow.
• ✓ Run validation before promotion.

Do not

• × Let one failed run rewrite future behavior.
• × Mix private logs, secrets, or hidden tests into reusable skills.
• × Treat the wiki as automatically true.
• × Let the skill file become a full incident archive.
• × Delete the lesson just because a proposed skill patch was rolled back.

The useful version is not mystical.

It is just normal engineering discipline applied to agent learning:

logs are evidence
wiki pages are lessons
skills are deployable behavior

Keep those three separate, and your agents can improve without turning every bad run into permanent superstition.

Sources

• WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolution
• Private Huecki radar artifact: content-research/radar/2026-08-29.md