# Your Agent Eval Is Too Short

Canonical URL: https://huecki.com/en/blog/agent-eval-too-short-trajectory/
Markdown URL: https://huecki.com/en/blog/agent-eval-too-short-trajectory.md
Language: English
Published: 2026-07-13
Updated: 2026-07-13
Author: Dominic Hückmann
Topic: AI Agent Workflows
- Agent topics: Agent Harnesses, Context Engineering, Agent Evals
- Tags: AI Agents, Evals, Developer Workflow, Agent Harness, AI Engineering
- Audience: developers building coding agents, small teams evaluating internal AI workflows, agent harness maintainers
- Concepts: trajectory evaluation, dense reward, agent reliability, failure recovery
Thesis: For long-running agents, final-answer scoring is too sparse. The useful eval unit is the trajectory: checkpoints, evidence, failure onset, recovery, and the human-visible artifact trail.
Content status: field-note

## Summary

A final pass/fail score hides the part of agent work that matters most: where the run started drifting, whether it noticed, and whether it recovered. The practical replacement is a trajectory eval with checkpoints, failure labels, and recovery metrics.

## Description

Long-running agents should not be judged only by the final answer. Score checkpoints, failure onset, recovery, and trajectory evidence before changing prompts or models.

## Body

## The short version

Most agent evals are still too short.

They ask whether the run ended correctly. That is useful, but for long-running terminal agents, coding agents, browser agents, and internal automations, the final answer is usually the least diagnostic part of the story.

The better question is:

> Where did the trajectory start to drift, and did the agent recover before the final artifact was damaged?

That is the shared signal in two new papers. [Long-Horizon-Terminal-Bench](https://arxiv.org/abs/2607.08964) uses dense grading for terminal tasks that run from minutes to hours. [Failure as a Process](https://arxiv.org/abs/2607.09510) analyzes coding-agent failures as onset, evolution, and recovery, not just a final red mark.

The workflow to steal is simple: score the path, not only the destination.

## Final pass/fail hides the useful failure

A final score can tell you that the agent failed. It rarely tells you why.

Did the agent misunderstand the task in step 2? Did it choose the wrong tool at step 9? Did it recover from a bad hypothesis and then break the build later? Did the final artifact pass only because the task was underspecified?

Those are different problems.

This matters because teams usually react to failed agent runs by changing the easiest visible thing: the prompt, the model, or the tool list.

Sometimes that is correct. Often it is guesswork.

If you have no trajectory labels, you cannot tell whether the agent needed a better model or just a checkpoint before it edited the wrong file.

## A trajectory eval you can run this week

Do this before the agent starts:

Use this template:

```txt
Task:
[one sentence]

Checkpoints:
1. The agent restates scope and forbidden moves.
2. The agent identifies required files, tools, or sources.
3. The agent produces a first artifact or plan with evidence.
4. The agent runs the relevant validator or review gate.
5. The agent reports final result, skipped checks, and residual risk.

Trajectory labels:
- onset: first step where the run became materially wrong or risky
- evolution: how the error spread or stayed contained
- recovery: whether the agent noticed, corrected, escalated, or kept going

Score:
- final artifact: pass / partial / fail
- checkpoint coverage: 0-5
- recovery: none / accidental / explicit / escalated
- evidence quality: claims only / logs / artifacts / reproducible checks
```

The important bit is not the exact labels. It is forcing yourself to review the run before editing the agent.

## Tiny prompt to steal

Use this after a failed or suspicious run:

```txt
Review this agent trajectory before proposing prompt, model, or tool changes.

Return:
1. First failure onset: step number, evidence, and why it mattered.
2. Evolution: whether the failure spread, got hidden, or stayed contained.
3. Recovery: what the agent did when evidence contradicted its path.
4. Missing checkpoint: the earliest checkpoint that would have caught it.
5. Smallest harness change: one rule, validator, memory reminder, or approval gate.
6. Regression case: one future task that should fail if this issue returns.
```

That prompt is deliberately boring. Good eval plumbing usually is.

## Where this fails

Trajectory scoring can become bureaucracy. Do not use it for every `sed` command, typo fix, or deterministic build step.

Use it when the run is long, expensive, ambiguous, risky, or repeated often enough that one failure teaches you something.

The trap is turning every trace into a courtroom transcript. You only need enough evidence to know what to change next.

Both sources are fresh preprints, so the operational template below should be treated as a practical synthesis, not a settled evaluation standard. Long-Horizon-Terminal-Bench demonstrates the value of dense intermediate rewards across 46 tasks. Failure as a Process provides the onset/evolution/recovery frame from 1,794 valid trajectories. Neither paper proves that every team should use exactly five checkpoints.

## Do this, not that

## Sources

- [Long-Horizon-Terminal-Bench](https://arxiv.org/abs/2607.08964)
- [Failure as a Process: An Anatomy of CLI Coding Agent Trajectories](https://arxiv.org/abs/2607.09510)
- [Import AI feed](https://importai.substack.com/feed)

## FAQ

### Is final task success useless for agents?

No. Final success still matters, but it is too sparse for long-running work. You also need checkpoints, drift labels, recovery evidence, and artifact-level proof.

### When is trajectory scoring worth the overhead?

Use it for long-running, ambiguous, risky, or repeated workflows. Skip it for short deterministic commands where ordinary tests and exit codes already explain enough.

### What should a team add first?

Start with five observable checkpoints, a simple failure-onset label, and one recovery metric before changing prompts, tools, or models.


## Source References

- [Long-Horizon-Terminal-Bench](https://arxiv.org/abs/2607.08964) (paper)
- [Failure as a Process](https://arxiv.org/abs/2607.09510) (paper)