Stop Treating LLM Judges Like Report Cards

Most LLM judges are used like report cards.

The agent finishes. The judge reads the answer. A score appears. Everyone pretends the number is operationally meaningful.

That is too late.

LLM-as-a-Verifier is interesting because it moves the judge closer to the control loop. The paper and repo are not just saying “ask a model to grade better.” They are saying: extract a finer signal from the verifier, then use that signal to select, monitor, and eventually train agent behavior.

That is the useful part for teams building agents.

Not the leaderboard claim by itself.

The useful idea is this:

A verifier score should steer execution, not decorate the postmortem.

86.5%

author-reported Terminal-Bench V2 result

78.2%

author-reported SWE-Bench Verified result

3

scaling axes: granularity, repeats, criteria

The Short Version

LLM-as-a-Verifier takes a normal judge pattern and changes the scoring step.

Instead of asking the model for one discrete score and trusting the sampled token, it looks at the probability distribution over score tokens and computes an expected value. That turns a coarse “probably 14 out of 20” into a continuous reward-like signal.

Then it scales the signal in three ways:

1. Use finer score granularity.
2. Repeat evaluations to reduce variance.
3. Split the task into criteria so one vague prompt does not decide everything.

That signal can be used for:

• selecting the best trajectory from several agent attempts,
• comparing two candidate solutions,
• tracking progress while a task is running,
• producing dense rewards for RL experiments.

The small-team translation is simple:

do not use a judge only after the agent is done

use a verifier to decide:
- keep going
- retry
- branch
- stop early
- escalate
- run deterministic checks

Verifier as control signal

semantic signal first; hard checks before trust

1. 01
   Task
2. 02
   Multiple trajectories
3. 03
   Verifier score
4. 04
   Select or retry
5. 05
   Deterministic checks
6. 06
   Ship or escalate

Why The Score Token Trick Matters

A classic LLM judge prompt throws away information.

It asks for a score, the model samples or chooses a token, and the system treats that token as the score.

For easy tasks, that might be fine. For agent trajectories, it is crude. Two solutions can both land on the same visible score even though the model’s probability mass says one is clearly stronger.

LLM-as-a-Verifier keeps more of that uncertainty.

If the possible score tokens are ordered, the verifier can compute the expected score under the model’s token probabilities. That gives a continuous signal instead of one lumpy integer.

This matters because agent work is often not a single answer. It is a trajectory:

• Did the agent inspect the right file?
• Did it test the right behavior?
• Did it confuse a symptom with the root cause?
• Did it stop after a green but irrelevant check?
• Did it create a fix that works only for the visible case?

A single final score hides too much.

The paper’s strongest engineering point is that verifier quality improves when you scale the verification process itself: more granular scores, repeated evaluations, and decomposed criteria. That sounds expensive, because it is. But it also gives you knobs.

You can spend verifier compute only where the decision matters.

What to steal

• Use continuous verifier scores for ranking, not just pass/fail reporting.
• Score whole trajectories when the task is agentic.
• Separate criteria such as root cause, execution evidence, safety, and final state.
• Repeat verification only for expensive or ambiguous decisions.
• Keep the verifier inside a budgeted harness, not as an unbounded second agent.

The Best First Use: Best-Of-N Agents

The lowest-friction use is best-of-N selection.

Run the agent several times. Let the verifier rank the resulting trajectories. Execute or present the best one only after the usual hard checks pass.

This is especially useful when generation is cheap enough and mistakes are expensive enough:

• coding agents fixing a failing test,
• research agents choosing a final synthesis,
• support agents drafting a resolution,
• browser agents planning a sequence of actions,
• data agents generating transformations or queries.

The repo ships this as llm_verifier.select(...), and the TurboAgent proxy applies the same idea to coding-agent responses: generate multiple candidates, score pairwise comparisons with a probabilistic pivot tournament, then return the selected response.

That is a better shape than “one rollout, one judge, one regret.”

But do not confuse selection with proof.

The selected trajectory is the most promising candidate according to the verifier. It still needs tests, diff review, tool output, schema checks, permission checks, and domain validation.

The verifier is a steering wheel.

It is not the finish line.

Progress Tracking Is The More Interesting Part

The paper and docs also describe progress tracking: score the trajectory at checkpoints.

That unlocks a different control loop.

Instead of waiting until the end, the harness can ask:

is this run becoming more likely to succeed?

If the score stays flat or falls, the harness can stop early, resample, branch, or ask for help. If the score rises but the hard verifier still fails, the harness can keep the trajectory but require a final repair pass.

For coding agents, that is practical.

You can imagine a run policy like this:

Every N tool steps:
1. Score progress against the task.
2. Check whether the agent has gathered new evidence.
3. Stop if progress is low and no new evidence appeared.
4. Branch if two candidate plans are plausible.
5. Require deterministic tests before final acceptance.

That is much more useful than a judge that wakes up after the PR is already written.

Judge vs verifier control

Report-card judge

• Scores only the final answer.
• Produces a number for dashboards.
• Uses one broad prompt.
• Can hide uncertainty in a sampled score token.
• Often treated as acceptance.

Control-loop verifier

• Scores candidate trajectories and checkpoints.
• Drives retry, branch, stop, or escalate decisions.
• Splits evaluation into criteria that match the task.
• Uses score-token probability mass as a continuous signal.
• Feeds hard checks before anything is trusted.

The Implementation Catch

The catch is not subtle.

The verifier backend needs token logprobs.

That means not every frontier model can be the verifier. The TurboAgent README says this plainly for Claude: Claude can generate candidates, but it cannot directly be the verifier because the Anthropic Messages API does not return token logprobs.

That does not make Claude useless in this setup. It means the roles split:

Claude / other strong model:
  generate candidate trajectories

Gemini via Vertex, DeepSeek, OpenAI-compatible vLLM, or another logprob backend:
  verify and score

The project docs also describe a two-stage workaround for logit-restricted models: let the closed model produce reasoning or a draft score, then route that through an open verifier that exposes logprobs.

Good to know.

But for production design, this should be a visible constraint in the architecture, not a surprise discovered after you build the harness.

Where Teams Will Misuse It

The predictable misuse is over-trusting the smoother number.

A continuous verifier score can look more scientific than a normal judge score. That does not make it ground truth.

Bad criteria still produce bad scores. A verifier can reward confident-looking but wrong work. Repeated evaluation can reduce variance without fixing a missing fact. Best-of-N can select the least bad trajectory from a weak candidate pool.

And if you use the same model family to generate, judge, and approve its own work, you should expect correlated blind spots.

So the practical pattern should look like this:

semantic verifier:
  which candidate seems best?
  is progress improving?
  is this trajectory worth more compute?

deterministic verifier:
  did tests pass?
  did the command succeed?
  does the schema match?
  is the permission valid?
  did the data actually change?

The semantic verifier decides where to spend attention.

The deterministic verifier decides what is allowed to happen.

Use it as a control signal

Do

• ✓ Use best-of-N selection for high-value agent runs.
• ✓ Track progress to stop hopeless trajectories earlier.
• ✓ Design criteria around the task: root cause, evidence, final state, safety.
• ✓ Require backend support for score-token logprobs before choosing the verifier model.
• ✓ Run tests, schemas, permissions, and domain checks after verifier selection.

Do not

• × Treat the verifier score as proof of correctness.
• × Let the same unbounded model generate, verify, approve, and deploy.
• × Spend repeated-evaluation budget on low-stakes tasks.
• × Hide verifier cost inside normal inference cost.
• × Use vague criteria and expect precise operational decisions.

A Small-Team Rollout Plan

Do not start with RL.

Start with a harness decision you already make manually.

For example: “Which of these three coding-agent attempts should I review?”

Then build the smallest useful verifier loop:

1. Save each candidate trajectory.
2. Define three criteria:
   - solved the actual task
   - used credible evidence
   - left the system in a safe final state
3. Run pairwise or best-of-N verification.
4. Send only the selected candidate into deterministic checks.
5. Log verifier score, final checks, human override, and outcome.
6. Turn disagreements into regression fixtures.

That last line is where the product improves.

The verifier is not valuable because it produces a prettier score. It is valuable because it creates a structured place to notice when your agent harness made the wrong control decision.

The Practical Takeaway

LLM-as-a-Verifier is worth reading because it shifts the conversation from “how do we grade the final answer?” to “how do we control agent execution with better uncertainty signals?”

That is the right direction.

But it does not remove the boring parts of reliable automation.

You still need tests. You still need tool evidence. You still need permission gates. You still need domain verifiers. You still need cost budgets.

Use the LLM verifier to decide which path deserves the next unit of compute.

Use deterministic checks to decide what is true enough to execute.

That split is the difference between an impressive judge demo and a useful agent system.

Sources

• LLM-as-a-Verifier: A General-Purpose Verification Framework
• arXiv HTML version
• LLM-as-a-Verifier GitHub repository
• LLM-as-a-Verifier documentation
• Progress tracking documentation
• Logit-restricted frontier models documentation
• TurboAgent repository
• Research artifact: content-research/deepresearch/llm-as-a-verifier-2026-08-24/report.md