Why test coding agents in hardened environments?

Because production controls such as restricted network access, scoped credentials, read-only filesystems, and non-root execution change success rate, cost, timeout behavior, and failure mode.

What is the smallest useful hardened eval?

Run the same representative task suite under baseline, limited network, read-only, scoped credential, and non-root policies, then label whether each failure was caused by the model, the harness, or an impossible policy.

Should security hardening be relaxed if an agent performs worse?

Not by default. First identify which tasks are genuinely blocked by policy, then improve task design, tooling, fixtures, or model routing before granting broader privileges.

Coding Agents Need Hardened Harness Evals

A practical workflow for testing coding agents under real enterprise constraints: scoped credentials, restricted egress, read-only filesystems, non-root users, and task-solvability labels.

The benchmark was too friendly

A lot of coding-agent testing still happens in a world that real companies do not run.

The agent has network access. The filesystem is writable. Credentials are broad enough to finish the task. Package installation works. The user is root, or close enough. When the benchmark passes, everyone quietly treats that as evidence the agent is ready.

The useful warning in Permission Denied is that this setup is too friendly. The paper evaluates coding agents on Terminal-Bench 2.1 under nested security policies derived from ordinary enterprise controls: scoped credentials, restricted egress, read-only filesystems, and non-root execution.

The practical takeaway is simple:

Evaluate the agent under the policy it will actually live inside.

coding agents evaluated in the paper

18.3

reported max success-point loss under strictest policy

167.3%

reported max cost inflation under strictest policy

What changes under policy

Hardening does not just lower a single score. It changes the shape of the run.

An agent that could previously curl documentation now needs local docs, a preloaded package cache, or an approved search tool. An agent that wrote helper files everywhere now needs a working directory contract. An agent that silently installed dependencies now needs an allowlist. An agent that assumed root needs to learn the difference between “task impossible” and “try a safer path.”

That distinction matters. If security policy blocks the only route to completion, the model did not fail. The task design did. If the task is still solvable but the agent loops until timeout, the harness or model needs work.

Permissive eval vs. hardened eval

Permissive

Agent can reach the internet whenever stuck.
Agent writes across the repo and temp space freely.
Credentials are broad enough to make the demo pass.
Failures collapse into pass/fail.

Hardened

Egress is restricted; allowed docs and package sources are explicit.
Writable paths are named; read-only areas catch hidden assumptions.
Credentials are scoped to the task and logged by capability.
Failures are labeled as model, harness, policy-blocked, timeout, or wrong solution.

A workflow to steal

Do not start by building a giant benchmark. Start with ten tasks your team actually wants the coding agent to do.

Boundary eval pass

before trusting a coding agent in a real repo

01

Pick tasks
02

Define policy levels
03

Run baseline
04

Run hardened
05

Label solvability
06

Route or fix

For each task, run the same agent under a small policy matrix:

Task:
Expected proof:

Policy level 0: normal dev sandbox
Policy level 1: restricted egress
Policy level 2: scoped credentials
Policy level 3: read-only repo except worktree
Policy level 4: non-root execution
Policy level 5: strict combined policy

Record per run:
- completed: yes/no
- tests or proof produced
- cost and wall time
- blocked action
- first bad workaround
- final failure type
- policy made task impossible: yes/no/unknown

The important part is not the exact levels. It is nesting. You want to know which control changes behavior, not only that the final strict mode is worse.

What to label

Policy-blocked: the task cannot be completed without a capability the policy intentionally denies.
Tooling gap: the task is valid, but the harness lacks a safe replacement path such as local docs, package cache, or scoped helper tool.
Model workaround failure: the agent keeps trying unsafe, impossible, or irrelevant actions after a denial.
Timeout inflation: the agent spends extra tokens and time without changing strategy.
Wrong solution under constraint: the agent produces a passing-looking patch that avoids the hard part.

Tiny policy matrix

Use this as the first version:

Coding-agent boundary matrix

Network:
- none
- docs-only
- package-registry-only
- unrestricted

Filesystem:
- repo read-only
- worktree write-only
- temp write allowed
- unrestricted

Credentials:
- none
- read-only token
- repo-scoped token
- deploy-capable token

Privilege:
- non-root
- root inside disposable container

External effects:
- disabled
- dry-run only
- explicit approval required

Then make the routing rule boring:

If the agent only succeeds with broad egress, broad credentials, or root, it is not ready for the default route.

That does not mean the agent is useless. It means the harness needs a narrower safe path, or the task class belongs behind an approval gate.

Where it fails

A hardened eval can also lie.

If the benchmark tasks were written for a permissive environment, strict policy may make half of them artificially impossible. If the task harness does not expose safe alternatives, agents get punished for obeying policy. If the evaluator treats “stopped because blocked” as equal to “made up a broken patch,” the score encourages worse behavior.

That is why task-solvability labels are the whole trick.

Hardened coding-agent evals

Do

✓ run the same tasks under nested policy levels
✓ separate policy-foreclosed tasks from model failures
✓ track cost, timeout, blocked actions, and wrong-solution modes
✓ add safe replacement paths before relaxing security
✓ route models by task class and policy tolerance

Do not

× trust permissive sandbox scores as production evidence
× treat every denial as a model failure
× give broad credentials just to preserve a benchmark score
× ignore cost inflation when hardening triggers long loops
× ship without a few impossible-task fixtures

The right question is not “which coding agent is best?”

It is:

Which agent can finish this task class inside this boundary, with proof, without grinding into expensive nonsense?

That is a much more useful evaluation than another friendly leaderboard run.

Sources

Permission Denied: Policy-Graded Evaluation of Coding Agents in Hardened Environments
CUADebug: Diagnosing and Repairing Computer-Use Agent Failures
ExplainBench: Evaluating Code Explanations from Agents
Private Huecki radar artifact: content-research/radar/2026-08-05.md