Is final task success useless for agents?

No. Final success still matters, but it is too sparse for long-running work. You also need checkpoints, drift labels, recovery evidence, and artifact-level proof.

When is trajectory scoring worth the overhead?

Use it for long-running, ambiguous, risky, or repeated workflows. Skip it for short deterministic commands where ordinary tests and exit codes already explain enough.

What should a team add first?

Start with five observable checkpoints, a simple failure-onset label, and one recovery metric before changing prompts, tools, or models.

Your Agent Eval Is Too Short

Long-running agents should not be judged only by the final answer. Score checkpoints, failure onset, recovery, and trajectory evidence before changing prompts or models.

The short version

Most agent evals are still too short.

They ask whether the run ended correctly. That is useful, but for long-running terminal agents, coding agents, browser agents, and internal automations, the final answer is usually the least diagnostic part of the story.

The better question is:

Where did the trajectory start to drift, and did the agent recover before the final artifact was damaged?

That is the shared signal in two new papers. Long-Horizon-Terminal-Bench uses dense grading for terminal tasks that run from minutes to hours. Failure as a Process analyzes coding-agent failures as onset, evolution, and recovery, not just a final red mark.

The workflow to steal is simple: score the path, not only the destination.

long-horizon benchmark tasks

1,794

valid coding-agent trajectories

63k+

execution steps manually annotated

Final pass/fail hides the useful failure

A final score can tell you that the agent failed. It rarely tells you why.

Did the agent misunderstand the task in step 2? Did it choose the wrong tool at step 9? Did it recover from a bad hypothesis and then break the build later? Did the final artifact pass only because the task was underspecified?

Those are different problems.

Final score vs. trajectory score

Final-only eval

Pass/fail at the end.
Treats failure as one event.
Rewards lucky final artifacts.
Pushes teams to swap models.

Trajectory eval

Milestones throughout the run.
Labels onset, evolution, and recovery.
Checks whether the path stayed sane.
Shows whether the harness, context, or recovery policy broke.

This matters because teams usually react to failed agent runs by changing the easiest visible thing: the prompt, the model, or the tool list.

Sometimes that is correct. Often it is guesswork.

If you have no trajectory labels, you cannot tell whether the agent needed a better model or just a checkpoint before it edited the wrong file.

A trajectory eval you can run this week

Do this before the agent starts:

Trajectory-first agent eval

make progress inspectable before the final artifact exists

01

Define checkpoints
02

Run agent
03

Log events
04

Label drift
05

Score recovery
06

Patch harness

Use this template:

Task:
[one sentence]

Checkpoints:
1. The agent restates scope and forbidden moves.
2. The agent identifies required files, tools, or sources.
3. The agent produces a first artifact or plan with evidence.
4. The agent runs the relevant validator or review gate.
5. The agent reports final result, skipped checks, and residual risk.

Trajectory labels:
- onset: first step where the run became materially wrong or risky
- evolution: how the error spread or stayed contained
- recovery: whether the agent noticed, corrected, escalated, or kept going

Score:
- final artifact: pass / partial / fail
- checkpoint coverage: 0-5
- recovery: none / accidental / explicit / escalated
- evidence quality: claims only / logs / artifacts / reproducible checks

The important bit is not the exact labels. It is forcing yourself to review the run before editing the agent.

Tiny prompt to steal

Use this after a failed or suspicious run:

Review this agent trajectory before proposing prompt, model, or tool changes.

Return:
1. First failure onset: step number, evidence, and why it mattered.
2. Evolution: whether the failure spread, got hidden, or stayed contained.
3. Recovery: what the agent did when evidence contradicted its path.
4. Missing checkpoint: the earliest checkpoint that would have caught it.
5. Smallest harness change: one rule, validator, memory reminder, or approval gate.
6. Regression case: one future task that should fail if this issue returns.

That prompt is deliberately boring. Good eval plumbing usually is.

Where this fails

Trajectory scoring can become bureaucracy. Do not use it for every sed command, typo fix, or deterministic build step.

Use it when the run is long, expensive, ambiguous, risky, or repeated often enough that one failure teaches you something.

Good first targets

repo-wide coding-agent tasks that touch multiple files
browser or desktop agents that act across several apps
research agents where the answer depends on source quality
support or ops agents that must escalate at the right moment
internal automations where silent partial success is dangerous

The trap is turning every trace into a courtroom transcript. You only need enough evidence to know what to change next.

Both sources are fresh preprints, so the operational template below should be treated as a practical synthesis, not a settled evaluation standard. Long-Horizon-Terminal-Bench demonstrates the value of dense intermediate rewards across 46 tasks. Failure as a Process provides the onset/evolution/recovery frame from 1,794 valid trajectories. Neither paper proves that every team should use exactly five checkpoints.