Agent Wiki Topic
Agent Evals
Measurable agent quality: benchmarks, review loops, playtests, regressions, and proof gates.
Agent-readable entry points
Start here
Your Agent Memory Test Is Probably Measuring the Wrong Thing
Most memory evals ask whether the agent got the final answer right. MemTrace suggests a sharper unit: one durable user fact tested across age, current state, earlier state, trajectory, and contradictory evidence. That turns memory from a vague feature into a small regression suite.
Markdown exportRelated pages
blog
Coding Agents Need Hardened Harness Evals
Permissive coding-agent benchmarks hide a boring production truth: security policy changes agent behavior. Small teams should run the same task suite under nested hardening levels and separate model failures from tasks the policy made impossible.
blog
Harness Engineering Is Systems Engineering for AI Agents
Harness engineering is the work of turning a probabilistic model into a controlled system. This field guide maps the twelve engineering surfaces around the model and shows how to evaluate and evolve them.
blog
Your Agent Harness Needs a Behavior Map
Harness Handbook points at a practical bottleneck in agent engineering: the behavior you want to change is scattered across prompts, state managers, tool calls, policy code, and tests. Build a behavior map before editing the harness.
blog
Your AI Agent Is Not Reflecting. It Is Defending Its First Answer
Asking one agent to reconsider its answer often produces a more confident defense of the same mistake. A bounded challenger-and-judge loop can create real alternatives, but only if disagreement, stopping, and judge bias are engineered explicitly.
blog
Your AI Agent Learned Something. Should It Be Allowed to Remember It?
An agent that writes a lesson into memory, a skill, a prompt, or its own code is deploying behavior into future runs. This guide shows how to put persistent changes through evidence, eval, approval, expiry, and rollback gates.
blog
The Perfect Automated AI Eval Stack Does Not Exist
The reliable eval system is not one automated judge. It is a closed loop that combines portable traces, deterministic invariants, narrow semantic judges, versioned production failures, adversarial tests, and human calibration.
blog
Your Agent Eval Is Too Short
A final pass/fail score hides the part of agent work that matters most: where the run started drifting, whether it noticed, and whether it recovered. The practical replacement is a trajectory eval with checkpoints, failure labels, and recovery metrics.
blog
Stop Asking Which Coding Model Is Best
The useful question is moving from which model is best to what your agent harness can change, measure, persist, and roll back.
blog
Your Agent Needs an Operating Contract, Not a Bigger Prompt
The serious agent pattern is no longer bigger prompts and more encouragement. It is an operating contract: measurable goal, bounded tools, context sources, verifier evidence, review notes, rollback path, and a skill update when the run teaches you something.
blog
Better AI Products Need Systems, Not One Agent
Better AI products come from improvement systems around the agent. This guide shows how to build one with deterministic checks, narrow scoring rubrics, private holdouts, calibrated judges, and promotion gates.
blog
Audit Local LLM Agents Like Runtimes
Local LLM agents can touch shells, files, browsers, credentials, memory, and messaging tools. Treat their runtime layer as source code worth auditing, then turn static findings into a manual review queue instead of automatic verdicts.
blog
Your Agent Memory Test Is Probably Measuring the Wrong Thing
Most memory evals ask whether the agent got the final answer right. MemTrace suggests a sharper unit: one durable user fact tested across age, current state, earlier state, trajectory, and contradictory evidence. That turns memory from a vague feature into a small regression suite.
tools
Is This Agentic Change Good?
This skill reviews branches that change future agent behavior. It collects local git evidence and judges skill collisions, context budget, safety risks, and runtime visibility before recommending a merge.
blog
Your Agent's Harness Is a Binary Now
Two 2026 papers from the same research lineage quietly retire prompt engineering as a discipline. The agent's system prompt is now a binary you can version, diff, and evolve with a 200-line loop. The four metrics that actually matter are not the ones your dashboard shows.
blog
AGENTS.md Is Not Context. It Is a Control Surface.
The surprising lesson from AGENTS.md benchmarks is not that context files are useless. It is that they change agent behavior, sometimes into more expensive and less useful work. Treat them as a control surface, not a repo manual.
blog
The Next Prompt Is Not a Prompt. It’s a Workflow.
Dynamic workflows move agent work from one chat prompt into inspectable orchestration: phases, subagents, evidence, budget, permissions, adversarial review, and stop conditions. The point is not more agents. The point is better control.
blog
Put an AI Slop Gate After Tests and Lint
Tests tell you whether behavior still works. Linters tell you whether code is syntactically and stylistically acceptable. An AI-slop gate catches the residue coding agents leave behind: fake comments, swallowed errors, any-casts, duplicated helpers, TODO stubs, and dead code.
blog
Debug AI Reward Functions Like Production Incidents
Bad reward functions should not be treated like prompt drafts. Treat them like production incidents: preserve traces, classify the failure, patch only the implicated logic, and rerun against the same controls.
blog
Your AI-Built UI Needs a Playtester, Not a Screenshot Review
AI-generated interfaces often look finished before they behave correctly. A GUI playtester loop uses a separate browser agent to interact with the artifact, record screenshots and action logs, turn broken flows into reproducible bug reports, and rerun the same script after repairs.
blog
Stop Judging AI Code by the Diff
Better AI coding is not mainly about better prompts. It is about the harness around the model: explicit contracts, separate builder and reviewer roles, evidence requirements, and a loop that turns failures into better specifications.
blog
Agents Don’t Need ‘Keep Going’. They Need Exit Conditions.
The useful lesson behind Claude Code /goal is not that agents can run forever. It is that long-running agent work needs an explicit, observable exit condition: what proves done, what stays in scope, and when to stop blocked.
blog
Don’t Benchmark the Model. Benchmark the Agent System.
Agent evals should not only ask whether the final answer looked good. A useful benchmark measures the whole agent system: skill routing, tool policy, evidence, outcomes, hard-fail safety cases, regressions, cost, and production drift.
blog
Give Your Agent Seatbelts, Not a Longer Prompt
When an agent keeps jumping from planning to editing to testing at the wrong time, the fix is not usually another paragraph of system prompt. Put the workflow into explicit states, give each state a tiny tool policy, and make phase changes visible.
blog
Agent harnesses should be specs, not hidden glue code
Natural-Language Agent Harnesses give a useful name to an important shift: the agent policy should be an inspectable document that a runtime executes, not invisible glue hidden inside controller code.
blog
Stop Asking AI to Critically Self-Check
Open-ended instructions like “critically self-check this” accidentally reward the model for producing criticism. The fix is not less review. It is calibrated review: explicit criteria, PASS_NO_CHANGE, evidence per finding, severity thresholds, and a tiny change budget.
blog
Agents Don’t Need Longer Prompts. They Need Harnesses.
The arXiv survey Code as Agent Harness names the next shift in agent engineering: code is not only what agents generate. It is becoming the executable, inspectable, stateful runtime that makes agents reliable.
blog
AGENTS.md is not enough: your coding agent needs a harness
A coding agent is not made reliable by one magic prompt. It needs a harness: AGENTS.md, skills, tool permissions, hooks, and evals that catch behavior drift.
blog
give every client project a tiny agent
The useful move is not one mega assistant for all client work. Give each client project a small, isolated agent with its own memory, tasks, preview URL habit, and boring daily standup.
blog
Prompt Decomposition: How to Break Down AI Tasks Properly
After context engineering comes decomposition: developers should stop putting everything into one prompt and instead split tasks into direct prompts, subtasks, pipelines, agent loops, or skills.
blog
The LLM-native developer needs more than prompts
The next developer skill is not writing clever prompts. It is building the operating system around LLMs: data quality, model versioning, evals, guardrails, incident response, review UX, and repo instructions agents can actually follow.
blog
Prompting Is Dead. Context Wins.
In 2026, good prompting is not about one magic sentence. The better approach is to curate context, define tools and schemas, set agent rules, and verify behavior with evals.
blog
Hermes Agent: Self-Review Instead of One-Shot Output
Hermes gets interesting when an agent does not only produce output, but reviews the run: execute, measure, critique, rewrite the skill, and test again. The loop pays off mainly for repeatable workflows.
blog
AI-first Architecture: Faster Decisions, Still in Control
AI-first architecture does not mean the model decides. It means AI generates options, finds risks, compresses context, and the team makes a traceable decision.