The Perfect Automated AI Eval Stack Does Not Exist

The button marked “evaluate everything” is a trap

The dream is understandable.

Send every production trace to a capable model. Ask it whether the agent did a good job. Put the score on a dashboard. Block releases when the number falls.

That sounds like an automated eval system.

It is actually an automated opinion system.

The missing piece is not a stronger judge. It is an operational loop that separates what software can verify from what a product team still has to discover.

The most useful recent evidence comes from Parlance Labs. The team took 100 real production traces from an apartment-leasing agent, worked with a domain expert to identify 39 failures, hid those annotations, and asked Braintrust Loop, Arize Alyx, LangSmith, Codex, Factory Droid, and Claude Code to analyze the traces.

The automated systems performed well. The best recovered 87.2% of the human-labeled failures. Every system also found valid issues the original review had missed.

Then came the important part: every system missed the same kind of problem.

The agent gave up at the first sales objection. It used Markdown in SMS. It talked over callers. It failed to hand off conversations that the business required a human to handle.

Those traces could look polite, coherent, and technically successful. What they lacked was the product’s intent.

100

real production traces

39

human-labeled failures

87.2%

best automated recall

The lesson is not that automated evaluation failed. Recovering most known failures from raw traces is useful.

The lesson is that automation has a boundary.

The trace contains the behavior. It does not contain the product’s intent.

What the reliable eval loop looks like

The best evaluation setup is not a single platform or judge. It is a loop that converts production evidence into durable tests.

The production eval loop

automate repetition; keep product judgment calibrated

1. 01
   Capture trace
2. 02
   Run checks
3. 03
   Sample review
4. 04
   Confirm failure
5. 05
   Add regression
6. 06
   Gate release

Each stage has a different job:

1. Capture enough evidence to reconstruct what happened.
2. Run cheap, exact checks before asking an LLM for an opinion.
3. Use narrow semantic judges where code cannot express the criterion.
4. Route ambiguous, novel, or high-risk cases to humans.
5. Turn every confirmed failure into a versioned regression case.
6. Test changes offline, then verify them again on canary traffic.

This is the architecture shared, in different forms, by the current documentation for Phoenix, Langfuse, Braintrust, and LangSmith.

Their interfaces differ. The useful pattern does not: online evaluation finds real failures; offline evaluation proves that a change fixes them; production monitoring checks whether the improvement survives contact with users.

Layer 1: make the trace portable

You cannot evaluate evidence you did not record.

For an agent, the final answer is only one event. A useful trace should include:

• user and system inputs
• model and prompt versions
• retrieved documents and their identifiers
• tool selection, arguments, output, and errors
• handoffs and guardrail decisions
• retries, loops, fallbacks, and recovery attempts
• final environment state
• latency, tokens, and cost by step

OpenTelemetry plus OpenInference is the pragmatic default because it keeps the instrumentation separate from the evaluation vendor. Phoenix accepts OTLP traces directly, and the broader ecosystem is converging on the same model of spans and events. The OpenAI Agents SDK similarly records generations, tool calls, handoffs, guardrails, and custom events in its tracing layer.

Portability matters. Your trace history is the raw material for future regression suites. It should not become inaccessible because the team changes orchestration frameworks or evaluation products.

Layer 2: code before judges

Many teams reach for an LLM judge too early.

If the requirement is exact, write an exact evaluator.

Choose the cheapest reliable evaluator

Deterministic check

• Did the JSON match the schema?
• Was the forbidden tool called?
• Does the database contain the booking?
• Did latency exceed the budget?

Semantic judge

• Was the answer genuinely helpful?
• Did the response handle the objection well?
• Was the explanation complete but concise?
• Did the agent understand the user's real goal?

Deterministic evaluators should own permissions, schemas, tool arguments, required handoffs, state transitions, citation existence, test execution, cost ceilings, and latency budgets.

These checks are cheap, explainable, and stable. They can usually run on every trace and block a release without philosophical debate.

The semantic judge begins where executable truth ends.

Layer 3: use narrow judges, not one quality score

“Rate this conversation from 1 to 10” is not an evaluation strategy.

One aggregate score hides which behavior changed and encourages the judge to substitute its own preferences for the product’s rules.

A better design uses one narrow grader per criterion:

• task completion
• factual grounding against supplied evidence
• required information coverage
• policy adherence
• objection handling
• tone or brand fit
• unnecessary tool use
• recovery quality after an error

Each grader needs observable anchors. A binary rubric is often more reliable than a vague ten-point scale. When a small ordinal scale is necessary, define what every value means and include borderline examples.

Require the judge to cite the message, tool result, or state transition supporting its decision. A verdict without evidence should be treated as low confidence.

Also version the judge. Its prompt, model, decoding configuration, rubric, and examples are production code. Changing any of them changes the measurement system.

Your judge needs an eval too

LLM judges are useful because they can evaluate qualities that ordinary assertions cannot express. They are dangerous because fluent explanations make weak measurements look authoritative.

G-Eval demonstrated that an LLM-based framework could align more closely with human judgments than older automatic metrics for summarization and dialogue. But its reported Spearman correlation for summarization was 0.514, not one, and the authors noted possible bias toward LLM-generated text.

A separate systematic position-bias study tested 12 judges over more than 100,000 evaluation instances. It found that answer order could influence the verdict and that the effect varied across judges and tasks.

This changes how judges should be deployed:

• Create a hidden calibration set labeled by domain experts.
• Measure precision and recall separately for every criterion.
• Test the same pair as A/B and B/A; inconsistent verdicts are abstentions.
• Blind the producing model and normalize superficial formatting where possible.
• Prefer a judge from a different model family than the generator.
• Recalibrate after changing the judge model, prompt, rubric, or product.
• Send judge-human disagreements to review instead of averaging them away.

Using another model family can reduce one source of self-preference. It does not remove position bias, verbosity bias, shared training priors, or misunderstanding of the rubric.

The judge is not ground truth. It is a classifier that must earn trust on labeled data.

Evaluate the outcome and the trajectory separately

An agent can reach the right answer through a dangerous process. It can also take a reasonable path and fail because one tool was unavailable.

Those runs should not receive the same diagnosis.

Outcome evaluators ask:

• Was the task completed?
• Was the final state correct?
• Did the user receive the required result?

Trajectory evaluators ask:

• Were the right tools selected with valid arguments?
• Did the agent follow required constraints?
• Did it repeat work or enter a loop?
• Did it notice contradictory evidence?
• Did it recover after a tool or reasoning failure?
• How much time and money did the path consume?

DeepEval exposes task-completion and process-oriented agent metrics. Ragas separates tool-call accuracy, tool-call F1, and agent-goal accuracy. These libraries are useful metric layers, but neither replaces a trace store, review workflow, and regression-data lifecycle.

The distinction also protects you from optimizing the wrong thing. A shorter trajectory is not automatically better. A longer run that verifies evidence and recovers safely may be preferable to a cheap run that succeeds by luck.

Build five datasets, not one benchmark

A random collection of prompts is not a production eval suite.

Maintain five pools with different purposes:

The five eval datasets

• Golden: stable core capabilities and business-critical happy paths.
• Regression: every confirmed production failure that must not recur.
• Adversarial: injection, leakage, permission, malformed-tool, and abuse cases.
• Canary: recent and novel production traffic that changes over time.
• Calibration: hidden human labels used to validate the evaluators themselves.

Each example should store more than input and ideal prose. Include the environment, expected final-state predicates, allowed and forbidden actions, criterion labels, severity, provenance, and relevant product version.

Start with 50 to 100 carefully reviewed cases. That is usually more useful than thousands of synthetic examples with uncertain labels.

Synthetic generation is valuable after a failure taxonomy exists. It can create variations of a known injection, handoff, or tool-argument problem. It cannot reliably invent the product requirement your team has not discovered yet.

Criteria drift is a feature of product work

The paper Who Validates the Validators? gives the recurring problem a useful name: criteria drift.

Teams need criteria to grade outputs. But grading outputs is how teams discover and refine those criteria.

That is why a perfectly written rubric rarely exists at the beginning. The apartment-leasing team did not merely forget to tell the automated systems about objection handling. Reading a conversation where a prospect walked away helped reveal that objection handling mattered.

The right response is not to remove humans from evaluation. It is to make their scarce attention compound.

Automate the work around judgment

Use when

• ✓ Sample representative and risky traces.
• ✓ Cluster similar suspected failures.
• ✓ Propose new criteria and regression cases.
• ✓ Find additional traces matching confirmed failures.
• ✓ Prepare focused disagreement queues.

Avoid when

• × Let the same agent propose and approve a rubric.
• × Infer business severity without product context.
• × Promote every suspected issue into ground truth.
• × Block high-stakes releases with an uncalibrated judge.
• × Replace domain review with a stable-looking average.

The human reviews a smaller, higher-value set. Every accepted decision improves the automated system and the regression suite.

A practical stack

For teams that value self-hosting and low lock-in, my default starting point is:

Instrumentation: OpenTelemetry + OpenInference
Control plane:   Arize Phoenix
Exact checks:    Python/TypeScript evaluators
Agent metrics:   DeepEval or Ragas
Semantic judge:  strong cross-family model, rubric-specific
Adversarial CI:  Promptfoo
Human loop:      weekly stratified annotation and disagreement review

Phoenix is not automatically the best product for every team. It is the strongest architectural starting point because it combines portable tracing, human/code/LLM evaluation, datasets, and experiments while remaining self-hostable.

Langfuse is a credible alternative for self-hosted product teams. Braintrust is attractive for an evaluation-first managed workflow and performed strongly in the Parlance comparison. LangSmith is particularly compelling for teams already committed to LangChain and now automates parts of the trace-to-evaluator workflow through Engine.

Do not select from feature pages alone. Run a bake-off using the same traces and reviewers. Measure:

• time from failed trace to regression test
• reviewer speed and evidence visibility
• evaluator versioning and exportability
• online sampling controls
• experiment comparison quality
• OTLP portability and raw-data access
• hosted cost and self-hosting burden

The best platform is the one that makes the learning loop routine, not the one with the longest list of built-in metrics.

Put evals into the release path carefully

Every prompt, model, tool, policy, or orchestration change should trigger the offline suite.

Compare the candidate against current production by criterion and task slice. Include latency, token use, and cost. Run nondeterministic cases more than once. Block immediately on critical deterministic failures. Block on semantic metrics only after the grader has demonstrated acceptable performance on the hidden calibration set.

Then shadow or canary the change on real traffic.

Online scoring should be risk-based:

• deterministic checks on as much traffic as practical
• semantic judges on a baseline sample
• heavier sampling for new versions, high-value workflows, anomalies, negative feedback, long traces, and high-cost runs
• automatic review queues for novel clusters and judge disagreement

Promptfoo adds a separate adversarial lane. Its red-team workflow can generate and execute attacks in CI and tag results with the git SHA and run identifier. Security testing should not be reduced to the same quality judge that rates helpfulness.

The 30-day rollout

Week 1: evidence

Instrument complete traces. Define five to ten critical business outcomes. Collect 50 representative and failed production examples. Add exact checks for state, permissions, tool use, latency, and cost.

Week 2: labels

Have domain experts label examples criterion by criterion. Freeze regression and calibration splits. Build narrow rubric judges. Measure their precision, recall, order consistency, and disagreement with humans.

Week 3: release gates

Run deterministic and regression suites in CI. Add adversarial tests. Compare candidate and production configurations. Gate critical invariants and calibrated regressions.

Week 4: production loop

Enable sampled online judging and risk-based oversampling. Establish a weekly 30- to 60-minute annotation review. Let automation propose new criteria and regression cases, but require a human to accept them.

The target is not 100% automation

The target is leverage.

A good eval system makes known failures cheap to catch forever. It detects regressions before users do. It shows where an agent’s trajectory broke. It turns production evidence into tests. It sends only novel and ambiguous cases to people who understand the product.

A bad eval system scores everything and explains nothing.

The perfect automated judge does not exist. The reliable automated eval loop can.

Sources and limitations

The central real-world comparison is Do Automated Evals Work? by Parlance Labs. It uses one dataset, one application, and 39 human-labeled failures. The authors explicitly warn against treating the results as a vendor ranking.

The judge recommendations draw on G-Eval, Judging the Judges, and Who Validates the Validators?. The implementation architecture is cross-checked against current documentation from Phoenix, Langfuse, Braintrust, LangSmith, DeepEval, Ragas, and Promptfoo.

Platform capabilities change quickly. Sampling rates, judge thresholds, data-retention requirements, and the final tool choice must be validated against the actual workload rather than copied as universal defaults.