What is harness engineering for AI agents?

Harness engineering is the design and operation of the executable system around a model: context construction, tools, state, permissions, orchestration, verification, recovery, observability, budgets, and evaluation.

How is harness engineering different from prompt engineering?

Prompt engineering changes instructions inside the model input. Harness engineering controls the full execution loop, including what enters context, which actions are allowed, how state persists, and how success is verified.

Is an agent framework the same as an agent harness?

No. A framework supplies primitives and defaults. The deployed harness is the concrete behavior created by your chosen model, prompts, tools, state policies, permissions, runtime, environment, and verifiers.

Can an AI agent safely improve its own harness?

It can propose and test candidate changes, but persistent promotion should remain behind independent regression tests, private holdouts, permission checks, canaries, versioning, and rollback.

Harness Engineering Is Systems Engineering for AI Agents

A field guide to the executable system around an AI model: context, tools, state, permissions, verification, recovery, tracing, evals, and controlled evolution.

The model proposes a tool call.

Something else decides whether the model can see the tool, whether the arguments are valid, whether the account has permission, whether the action needs approval, what the tool returns, whether the result counts as progress, and whether the loop should continue.

That “something else” is the harness.

For a long time, agent teams treated it as glue code. The model was the product. The prompt was the craft. The loop around both was plumbing.

That mental model is now backwards.

A 2026 wave of papers and production guides has started using harness engineering for the work of designing the executable system around a model. The phrase is new. The work is not. It combines runtime engineering, context management, tool and API design, state machines, authorization, observability, evaluation, and change control.

The useful equation is:

agent system = model + harness + environment

The model supplies probabilistic capability.

The harness turns proposals into controlled state transitions.

The environment supplies the consequences.

harness surfaces to engineer

primary and official sources reviewed

unit to evaluate: model + harness + environment

What an agent harness actually is

An agent harness is the executable control layer that repeatedly:

builds the model’s next context;
invokes the model under a role and budget;
parses the proposed response or action;
validates and authorizes it;
executes allowed tools;
records observations and state changes;
verifies progress or completion;
continues, repairs, escalates, or stops.

This is close to the definition used by Harness-Bench: the system layer that manages context, tools, state, constraints, permissions, tracing, and recovery.

It also gives us a better way to talk about capability. Harness-Bench ran 5,194 trajectories across model-harness pairings under shared task environments and budgets. The pairings differed substantially in completion, process quality, efficiency, and failure behavior.

So “Model X can do this task” is incomplete.

The defensible claim is:

This version of Model X, inside Harness Y, with Tool Set Z, Budget B, and Evaluator E achieved this result.

Four related engineering layers

Layer

Prompt engineering
Context engineering
Harness engineering
Evaluation engineering

What it controls

The instructions and examples included in one model input.
Which information enters the limited working set on every turn.
The complete executable control loop around the model.
The independent tasks, environments, graders, and statistics used to measure the system.

The last separation matters. An agent harness runs the agent. An evaluation harness runs experiments on the agent. If the mutable agent system also certifies its own success, you have built self-approval, not evidence.

The harness is an information-and-control system

The foundational agent loop is still simple:

The production agent loop

every arrow is a harness decision

01

Goal + policy
02

Context
03

Model proposal
04

Authorized action
05

Environment observation
06

Verification

But a real harness needs to keep three kinds of authority separate:

Epistemic authority: what evidence supports the answer?
Action authority: what may this run read, change, send, or spend?
Promotion authority: what may become durable memory, policy, skill, or configuration?

The model can propose across all three.

It should not grant itself any of them.

The twelve surfaces of harness engineering

1. Instructions and contracts

Prompts are good at expressing goals, roles, priorities, and heuristics. They are weak enforcement mechanisms.

If a requirement can be expressed as a deterministic predicate, enforce it as one. Authorization rules, schemas, output contracts, budgets, allowed paths, and destructive-action gates belong in code or policy outside the model.

The 2026 paper From Prompts to Contracts tested this distinction in a bounded enterprise workflow. Code-owned validators preserved source, routing, trace, and output contracts across model substitutions. Prompt-only enforcement allowed violations to reach the reader.

The case study is too narrow for a universal performance claim. The engineering lesson is still sound:

use prose for judgment
use code for invariants

2. Context construction

Context is not storage. It is working memory with an attention cost.

Anthropic’s context-engineering guidance recommends finding the smallest high-signal set of tokens, loading information just in time, compacting long histories, and leaving structured notes for long-running work.

A harness decides:

which instructions and tool schemas are loaded;
which files, messages, and memories are retrieved;
what is summarized or discarded;
how provenance survives compaction;
where untrusted content is marked;
which secrets and tenant data must be removed;
which parts remain stable for prompt caching.

A larger context window does not remove this job. Lost in the Middle found that relevant information could become harder to use when positioned inside long inputs. Capacity is not attention allocation.

3. Tools and affordances

An agent tool is a contract between deterministic software and a nondeterministic caller.

That changes interface design. A conventional API can expect a developer to read documentation, join data, select the right endpoint, and recover from errors. A model has to infer all of that from names, descriptions, schemas, examples, and returned context.

Anthropic’s tool-design guide recommends distinct tool purposes, strict inputs, high-signal results, useful namespacing, and an eval-driven improvement loop. It also warns that more tools are not automatically better. Overlap creates selection errors and consumes context.

Every consequential tool should declare:

purpose | scopes | read/write class | side effects | reversibility
input schema | output schema | timeout | retry | idempotency | dry run

MCP can standardize discovery and invocation. It does not make a tool trustworthy, correctly scoped, or safe to execute.

4. State and memory

“Memory” is too broad to be one bucket. Separate at least:

turn state: current messages and observations;
run state: plan, pending actions, budgets, checkpoints;
project state: artifacts and decisions needed to resume;
durable memory: facts or preferences retrieved across runs;
policy state: skills, prompts, rules, or code that alter future behavior.

These need different retention, provenance, review, expiry, and rollback rules.

A transcript is raw evidence. A summary is a lossy derived artifact. A candidate lesson is not yet policy. Once a remembered lesson changes future tool use, it has become part of the deployed harness.

5. Planning, routing, and delegation

Plans can remain implicit, become explicit mutable artifacts, or be split across planner and executor roles. Routing can be deterministic or model-selected. Multiple agents can run sequentially, concurrently, through handoffs, or under a manager.

None of these patterns is inherently more mature.

Add another agent when it creates a measurable benefit:

isolated context;
real parallelism;
different tools or permissions;
independent candidate generation;
separation of proposer and verifier;
a genuine security boundary.

If several agents share the same model, evidence, and prompt family, they may only reproduce the same error more expensively.

6. Environment and execution

Agents act inside environments: filesystems, browsers, databases, APIs, terminals, or desktops. The environment determines which actions are possible, observable, reversible, and gradeable.

Production harnesses need isolation, initialization, snapshots, scoped credentials, network policy, resource quotas, and reset logic.

The environment is also part of measured capability. Anthropic found that infrastructure error rates in one coding-agent evaluation fell from 5.8% under strict resource enforcement to 0.5% without caps. A benchmark can accidentally measure container pressure instead of intelligence. Infrastructure-noise study

7. Verification and feedback

Verification is where a harness pays rent.

Prefer evidence in this order:

executable environment-state checks;
deterministic schemas and validators;
independent reference data;
calibrated model graders with explicit rubrics;
self-critique as a weak diagnostic signal.

An agent saying “the flight is booked” is not the outcome. A reservation in the database is the outcome. Anthropic’s agent-eval guide makes this distinction explicit.

Progress checks and completion checks should also be separate. Passing one test can justify the next step. It does not prove the user’s objective is complete.

8. Recovery and stopping

A harness needs named policies for transient errors, invalid arguments, semantic failures, repeated actions, no-progress loops, corrupted state, and exhausted budgets.

Useful controls include:

classified retries with backoff;
state refresh before another attempt;
repeated-action and no-progress detectors;
checkpoint and rollback;
alternative-tool or strategy branches;
escalation with a compact evidence package;
hard caps on turns, tokens, time, cost, and side effects;
stopping based on verified outcome rather than model confidence.

Infinite reflection is not recovery. Without new evidence, another reasoning turn can become a more articulate defense of the current mistake.

9. Permissions and security

The minimum posture is capability-based least privilege.

Give each run and role only the tools, paths, accounts, network destinations, and credentials it needs. Separate reading from writing. Gate external, destructive, expensive, or irreversible actions. Bind tokens to their intended resources. Treat external content as untrusted data, not higher-priority instructions.

Test the actual attack surface:

indirect prompt injection in tool results;
malicious tool metadata;
confused-deputy and token-audience failures;
secret leakage through traces or handoffs;
cross-user context leakage;
memory poisoning;
unsafe retries and duplicate side effects;
sandbox escape;
denial of wallet.

AgentDojo provides 629 security cases across realistic tool tasks. HarnessAudit adds a crucial insight: a correct final answer can hide unauthorized access or information-flow violations earlier in the trajectory.

10. Observability and replay

A useful trace records enough to reconstruct the run:

model + harness version
context sources and hashes
model calls and sampling settings
tool requests, authorization decisions, results, timing
state checkpoints and environment changes
handoffs, guardrails, retries, costs
verifier output and final outcome

This supports debugging, replay, counterfactual comparison, regression localization, failure clustering, and audit.

It also creates a sensitive dataset. Trace access, retention, sampling, and redaction are security decisions, not observability afterthoughts.

11. Budgets and economics

The harness decides how often context is replayed, when it is compacted, how much tool output returns, how many branches run, when retries stop, and how failed spend is capped.

Measure:

cost per attempt and successful task;
median and tail latency;
tokens by role and loop stage;
tool and infrastructure cost;
retries and abandoned spend;
human review time;
safe success per euro, not tokens in isolation.

The vendor-authored paper The Harness Effect reported 38% fewer tokens, 41% lower blended cost, and 44% lower median wall time on 22 tasks after swapping the orchestration layer while holding six models constant. The workload is too small to universalize the percentages. It is enough to show that the harness can be a first-order economic variable.

12. Change control and maintainability

Harness behavior rarely lives in one file. It is distributed across prompts, tool wrappers, state reducers, middleware, policy branches, runtime defaults, and tests.

That makes behavior localization part of the engineering discipline. A useful harness has a map from observable behavior to implementation surfaces and proof. It also has versioned configuration, component ownership, replayable failures, change manifests, regression suites, private holdouts, canaries, and rollback.

Every change should make a falsifiable prediction:

Changing X should reduce failure Y on task slice Z
without worsening safety A, reliability B, or cost C.

If you cannot state what evidence would reject the change, you are not engineering the harness. You are decorating it.

How to evaluate a harness

The evaluated unit should be versioned explicitly:

model + prompt + context policy + tools + middleware + state policy
+ permissions + runtime + environment + verifier + budgets

Then test it in layers.

Harness evaluation stack

a passing demo is only the first layer

01

Component tests
02

Scenario tasks
03

Trajectory checks
04

Adversarial cases
05

Repeated trials
06

Private holdout

The minimum metric set is broader than task success:

task success and safe success;
policy violations and unintended side effects;
reliability across repeated trials;
median and tail cost and latency;
tool-call and recovery rates;
human escalation rate;
infrastructure failures.

tau-bench introduced pass^k to expose reliability across repeated interactive trials. ToolSandbox tests state dependencies and intermediate milestones. Terminal-Bench uses isolated terminal environments, human-written solutions, and executable tests.

Each catches failures that a one-shot answer benchmark cannot see.

The workflow to steal

Nine steps for harness engineering

Define the outcome in environment state and write the authority envelope.
Build the smallest executable loop with a small tool set and hard budgets.
Create representative, adversarial, and repeated-trial evals before optimizing.
Trace the complete trajectory so failures are attributable to one surface.
Classify failures before editing; do not answer every incident with more prompt text.
Change one behavior surface at a time and record a falsifiable prediction.
Run regression, adversarial, cost, safety, and private holdout checks.
Canary the change under limited authority and watch tail failures.
Promote, expire, or roll back based on observed evidence.

What not to do

Harness engineering

Build

✓ Small high-signal contexts with provenance.
✓ Narrow tools with explicit side effects and scopes.
✓ Environment-state verification and full trajectory traces.
✓ Bounded, versioned, reversible behavior changes.

Avoid

× Prompt accretion after every incident.
× Hundreds of overlapping tools or one ambient-admin god tool.
× Self-reported completion and one-run benchmarks.
× Production memory or policy that rewrites itself without promotion gates.

Framework adoption is another common trap. LangGraph, AutoGen, Semantic Kernel, OpenAI Agents SDK, and Microsoft Agent Framework provide useful primitives and defaults. They do not answer your product’s questions about authority, evidence, state, recovery, or risk.

A framework is not a harness architecture.

It is one input to it.

Should the harness improve itself?

Several 2026 papers treat harness code as an optimization surface.

Agentic Harness Engineering exposes editable components, distills failure traces, and attaches predictions to candidate changes. Self-Harness mines model-specific weaknesses and accepts changes after regression tests. TTHE evolves harness state from unlabeled test-time traces.

The results are promising.

They are not settled.

A new matched-budget study, Rethinking the Evaluation of Harness Evolution, found that automatic harness evolution did not consistently outperform simpler test-time scaling and showed limited held-out generalization in its experiments.

That changes the production recommendation.

Let agents diagnose failures, generate candidate edits, and run sandboxed experiments. Do not let the same system promote its preferred change directly into production.

Keep these outside the evolving layer:

immutable authority policy;
independent regression and adversarial evals;
private holdouts;
canary controls;
version history and rollback;
final promotion authority.

The interesting question is not whether an agent can rewrite its harness.

It is what evidence must exist before that rewrite governs the next user.

A maturity model

Level 0 — Prompt demo
one prompt, broad tools, manual inspection

Level 1 — Instrumented loop
explicit tools, budgets, traces, outcome checks

Level 2 — Controlled system
scoped permissions, isolation, checkpoints, recovery, repeated evals

Level 3 — Managed product
behavior map, versioning, private holdouts, adversarial tests, canaries

Level 4 — Adaptive platform
evidence-driven candidate changes, independent promotion, automatic rollback

Most teams should reach Level 2 or 3 before they attempt Level 4.

Harness engineering is not a new name for prompting

The term is still young. There is no stable industry boundary and no standards body defining the job. Some sources include the sandbox and eval system inside the harness. Others place them next to it. Some focus on coding agents. Others include personal assistants, browser agents, enterprise workflows, and multi-agent systems.

That ambiguity does not make the idea useless.

It points to the actual shift:

The model is no longer the complete engineering object.

The complete object is a probabilistic component inside a versioned, observable, permissioned, and verifiable system.

The strongest harness is not the one with the most agents, tools, memory, context, or autonomy. It is the smallest system that gives the model enough information and affordance to complete the task while keeping authority, verification, and rollback outside the model’s discretion.

Model intelligence is rented.

System behavior is engineered.