# Your Agent Needs a Wiki Before It Writes Better Skills Canonical URL: https://huecki.com/en/blog/wikiskill-agent-wiki-before-skills/ Markdown URL: https://huecki.com/en/blog/wikiskill-agent-wiki-before-skills.md Language: English Published: 2026-08-29 Updated: 2026-08-29 Author: Dominic Hückmann Topic: AI Agent Workflows - Agent topics: Context Engineering, Agent Evals - Tags: AI Agents, Agent Skills, Agent Memory, Developer Workflow, Self-Improving Agents - Audience: agent builders, developers using coding agents, AI product engineers, teams maintaining reusable workflows - Concepts: wikiskill, agent-skills, agent-memory, skill-evolution, knowledge-base, eval-gates Thesis: Agents learn more safely when raw traces, durable knowledge, and executable skills are separate layers. Content status: field-note ## Summary A practical developer reading of WikiSkill: do not let agents jump from one messy trace straight into a permanent skill. Put a persistent wiki layer between raw experience and executable instructions, then promote only changes that survive validation. ## Description WikiSkill points to a practical pattern for agent builders: keep raw runs, compile repeated lessons into a wiki, and only then update reusable skills behind an eval gate. ## Body ## The ELI5 version Imagine a junior developer keeps a notebook while learning a codebase. On Monday, they break the build because they edited a generated file. On Tuesday, they break it again because they forgot the migration command. On Wednesday, they finally write a clear checklist: ```txt For API client changes: 1. Edit the schema, not the generated client. 2. Run the generator. 3. Run the API contract tests. 4. Check the generated diff. ``` That checklist is useful because it is not just a random note from one bad day. It is a repeated lesson, cleaned up into a reusable procedure. That is the practical idea behind [WikiSkill](https://arxiv.org/abs/2608.27454). Do not let an agent jump straight from messy experience to permanent behavior. First, keep the raw experience. Then compile the repeated lessons into a wiki. Then update the skill. ## Why this matters Agent skills are useful because they package practical know-how: instructions, workflows, scripts, examples, and "when to use this" rules. The hard part is keeping those skills good. If a human writes every skill by hand, the system learns slowly. If the agent rewrites its own skills after every run, the system can learn nonsense quickly. WikiSkill is interesting because it inserts a middle layer. The paper separates the workspace into three layers: - **Raw layer:** immutable execution traces from agent runs. - **Wiki layer:** persistent pattern pages, logs, and skill-impact history. - **Skills layer:** executable procedural instructions that future agents use. That separation is the part developers should steal. Raw traces answer: what happened? The wiki answers: what have we learned across runs? Skills answer: what should the next agent do differently? ## The trap: one run is not wisdom After a failed run, an agent can always invent a lesson. Sometimes the lesson is right: ```txt Do not edit generated files directly. ``` Sometimes it is too broad: ```txt Never touch generated files. ``` Sometimes it is actively harmful: ```txt If the build fails, skip generated files in tests. ``` This is why a wiki layer matters. It gives the system somewhere to collect repeated evidence before changing the instructions future agents inherit. The last point is subtle and important. In WikiSkill, candidate skill changes are gated by validation. If a proposed skill hurts performance, the skill can be rolled back. But the wiki persists. The system still remembers that the attempted fix failed, so it does not keep proposing the same bad edit. That is closer to how good engineering teams learn. You can revert a bad code change without deleting the incident write-up. ## What the paper found WikiSkill evaluates this idea across five benchmarks: mathematical reasoning, web search, spreadsheet manipulation, long-context document question answering, and interactive embodied tasks. The authors report that WikiSkill improves over no-skill baselines in most model-benchmark settings and outperforms prior skill-evolution methods in their experiments. Two findings are especially useful for builders: First, skills and model scale are complementary. The paper reports larger gains for stronger Qwen models: 12.3%, 17.5%, and 23.9% average improvement for 4B, 9B, and 27B models respectively. Second, skill discovery and skill execution are not the same ability. In one reported ALFWorld result, a 9B model did better with a skill evolved by a 27B model than with its own self-evolved skill. Translated into engineering terms: ```txt Use strong agents to discover workflows. Let cheaper agents execute the workflows when they can. ``` That is not guaranteed for every task. But it is a good architecture hint. ## A practical version for your team You do not need the full research setup to copy the useful pattern. Start with three folders in your agent workspace: ```txt agent-learning/ raw/ 2026-08-29-run-184.md wiki/ patterns/ generated-files.md migration-order.md skill-impact.md skills/ api-client-change/ SKILL.md PURPOSE.md ``` Keep raw traces append-only. They are evidence, not polished truth. Use wiki pattern pages for repeated lessons: ```md # Generated Client Changes Observed in: - run-184: direct edit disappeared after generator ran - run-211: tests passed locally but CI regenerated client Pattern: Generated clients are outputs. Durable changes must start from the schema. Useful procedure: 1. Edit `schemas/order-api.yaml`. 2. Run `npm run generate:order-client`. 3. Run `npm test -- order-client`. 4. Inspect both schema and generated diff. Known exceptions: - Fixture snapshots under `test/generated-fixtures/` are manually reviewed. ``` Then make the skill short: ```md # API Client Change Skill Use when changing generated API clients. Procedure: 1. Find the source schema before editing generated files. 2. Change the schema or generator input. 3. Run the generator. 4. Run contract tests. 5. Report schema diff, generated diff, and test command. ``` The wiki can be verbose. The skill should be compact. ## The release gate Do not promote every suggested skill edit. Use a small validation set. For a coding agent, that can be five saved tasks: - generated client change; - migration change; - UI route update; - failing test fix; - dependency upgrade. When the agent proposes a skill update, run those tasks or replay fixtures. Accept the update only if it improves the target case and does not regress the others. This is the part many self-improving agent demos skip. The agent should be allowed to propose learning. Promotion is a separate operation. ## Where this can go wrong A wiki can become junk. If it stores every thought, it becomes another messy chat log. If it stores untrusted content without provenance, it can preserve prompt injections or stale assumptions. If it keeps only successes, it loses the negative evidence that prevents repeated bad edits. The useful version is not mystical. It is just normal engineering discipline applied to agent learning: ```txt logs are evidence wiki pages are lessons skills are deployable behavior ``` Keep those three separate, and your agents can improve without turning every bad run into permanent superstition. ## Sources - [WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolution](https://arxiv.org/abs/2608.27454) - Private Huecki radar artifact: `content-research/radar/2026-08-29.md` ## FAQ ### What is WikiSkill in simple terms? WikiSkill is a framework where agents keep raw execution traces, compile repeated lessons into a persistent wiki, and use that wiki to propose better reusable skills. ### Why not write skills directly from failed runs? One run can produce a useful clue, but it can also produce a superstition. A wiki layer lets the system compare patterns across runs before changing future behavior. ### What is the first thing a small team should copy? Create three folders: raw traces, wiki patterns, and skills. Promote a skill change only when a validation task improves and no regression appears. ## Related - Shows the small-team trace-to-skill loop that WikiSkill makes more systematic.: failed-agent-runs-should-become-skills - Explains why durable agent learning needs evidence, scope, approval, and rollback.: should-ai-agent-remember-what-it-learned - German companion on knowledge flywheels for agent runs.: dein-ai-agent-lernt-nichts-aus-seinen-runs ## Source References - [WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolution](https://arxiv.org/abs/2608.27454) (paper)