# Coding Agents Need Hardened Harness Evals

Canonical URL: https://huecki.com/en/blog/coding-agents-need-hardened-harness-evals/
Markdown URL: https://huecki.com/en/blog/coding-agents-need-hardened-harness-evals.md
Language: English
Published: 2026-08-05
Updated: 2026-08-05
Author: Dominic Hückmann
Topic: AI Agent Security
- Agent topics: Agent Harnesses, Agent Security, Context Engineering, Agent Evals
- Tags: AI Agents, Coding Agents, Agent Evals, Security, Developer Workflow
- Audience: engineering leads piloting coding agents, platform teams, security reviewers, developers building agent harnesses
- Concepts: coding-agent-evals, sandbox-policy, Boundary-Bench, Terminal-Bench, least-privilege
Thesis: A coding agent is not validated until it succeeds under the permissions, filesystem, network, credential, and privilege limits it will actually face.
Content status: field-note

## Summary

Permissive coding-agent benchmarks hide a boring production truth: security policy changes agent behavior. Small teams should run the same task suite under nested hardening levels and separate model failures from tasks the policy made impossible.

## Description

A practical workflow for testing coding agents under real enterprise constraints: scoped credentials, restricted egress, read-only filesystems, non-root users, and task-solvability labels.

## Body

## The benchmark was too friendly

A lot of coding-agent testing still happens in a world that real companies do not run.

The agent has network access. The filesystem is writable. Credentials are broad enough to finish the task. Package installation works. The user is root, or close enough. When the benchmark passes, everyone quietly treats that as evidence the agent is ready.

The useful warning in [*Permission Denied*](https://arxiv.org/abs/2608.02670) is that this setup is too friendly. The paper evaluates coding agents on Terminal-Bench 2.1 under nested security policies derived from ordinary enterprise controls: scoped credentials, restricted egress, read-only filesystems, and non-root execution.

The practical takeaway is simple:

```txt
Evaluate the agent under the policy it will actually live inside.
```

## What changes under policy

Hardening does not just lower a single score. It changes the shape of the run.

An agent that could previously `curl` documentation now needs local docs, a preloaded package cache, or an approved search tool. An agent that wrote helper files everywhere now needs a working directory contract. An agent that silently installed dependencies now needs an allowlist. An agent that assumed root needs to learn the difference between "task impossible" and "try a safer path."

That distinction matters. If security policy blocks the only route to completion, the model did not fail. The task design did. If the task is still solvable but the agent loops until timeout, the harness or model needs work.

## A workflow to steal

Do not start by building a giant benchmark. Start with ten tasks your team actually wants the coding agent to do.

For each task, run the same agent under a small policy matrix:

```txt
Task:
Expected proof:

Policy level 0: normal dev sandbox
Policy level 1: restricted egress
Policy level 2: scoped credentials
Policy level 3: read-only repo except worktree
Policy level 4: non-root execution
Policy level 5: strict combined policy

Record per run:
- completed: yes/no
- tests or proof produced
- cost and wall time
- blocked action
- first bad workaround
- final failure type
- policy made task impossible: yes/no/unknown
```

The important part is not the exact levels. It is nesting. You want to know which control changes behavior, not only that the final strict mode is worse.

## Tiny policy matrix

Use this as the first version:

```txt
Coding-agent boundary matrix

Network:
- none
- docs-only
- package-registry-only
- unrestricted

Filesystem:
- repo read-only
- worktree write-only
- temp write allowed
- unrestricted

Credentials:
- none
- read-only token
- repo-scoped token
- deploy-capable token

Privilege:
- non-root
- root inside disposable container

External effects:
- disabled
- dry-run only
- explicit approval required
```

Then make the routing rule boring:

```txt
If the agent only succeeds with broad egress, broad credentials, or root, it is not ready for the default route.
```

That does not mean the agent is useless. It means the harness needs a narrower safe path, or the task class belongs behind an approval gate.

## Where it fails

A hardened eval can also lie.

If the benchmark tasks were written for a permissive environment, strict policy may make half of them artificially impossible. If the task harness does not expose safe alternatives, agents get punished for obeying policy. If the evaluator treats "stopped because blocked" as equal to "made up a broken patch," the score encourages worse behavior.

That is why task-solvability labels are the whole trick.

The right question is not "which coding agent is best?"

It is:

```txt
Which agent can finish this task class inside this boundary, with proof, without grinding into expensive nonsense?
```

That is a much more useful evaluation than another friendly leaderboard run.

## Sources

- [Permission Denied: Policy-Graded Evaluation of Coding Agents in Hardened Environments](https://arxiv.org/abs/2608.02670)
- [CUADebug: Diagnosing and Repairing Computer-Use Agent Failures](https://arxiv.org/abs/2608.02643)
- [ExplainBench: Evaluating Code Explanations from Agents](https://huggingface.co/papers/2607.26451)
- Private Huecki radar artifact: `content-research/radar/2026-08-05.md`

## FAQ

### Why test coding agents in hardened environments?

Because production controls such as restricted network access, scoped credentials, read-only filesystems, and non-root execution change success rate, cost, timeout behavior, and failure mode.

### What is the smallest useful hardened eval?

Run the same representative task suite under baseline, limited network, read-only, scoped credential, and non-root policies, then label whether each failure was caused by the model, the harness, or an impossible policy.

### Should security hardening be relaxed if an agent performs worse?

Not by default. First identify which tasks are genuinely blocked by policy, then improve task design, tooling, fixtures, or model routing before granting broader privileges.


## Related

- Explains why model benchmarks are only input evidence, not deployment validation.: benchmark-scores-are-not-agent-validation
- Shows the same permission-label habit for tool-calling agents.: mcp-tools-permission-labels
- Covers command-trace risk once agents can run shell workflows.: coding-agent-command-composition-risk

## Source References

- [Permission Denied: Policy-Graded Evaluation of Coding Agents in Hardened Environments](https://arxiv.org/abs/2608.02670) (paper)
- [CUADebug: Diagnosing and Repairing Computer-Use Agent Failures](https://arxiv.org/abs/2608.02643) (paper)
- [ExplainBench: Evaluating Code Explanations from Agents](https://huggingface.co/papers/2607.26451) (paper)