# Better AI Products Need Systems, Not One Agent

Canonical URL: https://huecki.com/en/blog/agent-is-not-the-product-fitness-function-is/
Markdown URL: https://huecki.com/en/blog/agent-is-not-the-product-fitness-function-is.md
Language: English
Published: 2026-06-26
Updated: 2026-07-16
Author: Dominic Hückmann
Topic: AI Agent Workflows
- Agent topics: Agent Evals, Agent Harnesses, LLM-native Engineering, Context Engineering
- Tags: AI Agents, Agent Evals, Self-Improvement, Coding Agents, AI Engineering
- Audience: developers building agent workflows, small teams adopting coding agents, AI product engineers
- Concepts: agent-evals, agent-harnesses, llm-native-engineering
Thesis: The product advantage is the whole improvement system around the agent, not the agent alone.
Content status: field-note

## Summary

Better AI products come from improvement systems around the agent. This guide shows how to build one with deterministic checks, narrow scoring rubrics, private holdouts, calibrated judges, and promotion gates.

## Description

A practical guide to self-improving agent systems: concrete evals, scoring rubrics, hard-fail rules, private holdouts, judge calibration, and safe promotion gates.

## Body

The common mistake in AI products is to bet everything on one system:

```txt
Pick a strong model.
Wrap it in a product.
Hope it gets better.
```

That is not how the interesting products improve.

The better pattern is a system around the model: agents that propose changes, evaluators that judge them, tests that catch regressions, traces that explain failures, and promotion rules that decide what reaches users.

The seductive version of self-improving agents sounds like this:

```txt
Let the agent evolve itself until we get the winner.
```

That is the wrong mental model.

The useful version is colder:

```txt
Generate many candidate changes.
Run each candidate in a sandbox.
Score them with a trusted fitness function.
Keep what survives.
Promote only under constraints.
```

The agent is not the product. The improvement system is.

This is the lesson I would take from AlphaEvolve, FunSearch, SICA, and CodeEvolve. The frontier is not "one clever agent." It is **agent + evaluator + search loop + governance**.

## The research pattern

DeepMind's [AlphaEvolve](https://arxiv.org/abs/2506.13131) is the cleanest public example. It uses LLMs to propose code changes, then evaluates those programs with automated metrics. DeepMind describes the loop as Gemini models proposing candidate programs, evaluators verifying and scoring them, and a program database deciding what feeds future prompts.

That distinction matters. The LLM is not trusted because it sounds smart. It is useful because it can create candidates for a product-quality system.

AlphaEvolve is also a good guardrail against hype. Its strongest results are in domains where solutions can be described as code and automatically verified. DeepMind reports a production scheduling heuristic for Borg, a verified Verilog simplification for TPU hardware, Gemini kernel speedups, FlashAttention speedups, and new mathematical constructions. Those are not vague vibes. They are measurable artifacts.

FunSearch, the earlier Nature work, makes the same shape obvious: pair an LLM with a systematic evaluator and search over programs. SICA pushes in a more self-referential direction: the agent edits its own codebase and improves on benchmarks. CodeEvolve shows the open-source engineering version: island populations, LLM-driven variation, structured diffs, sandboxed execution, time and memory limits, and fitness extraction.

Different systems. Same spine.

## Not every agent should evolve

The sentence "all agentic systems should self-evolve" is too broad.

A calendar assistant does not need an evolutionary loop. A support triage bot probably does not need one on day one. A research assistant may benefit more from trace logging, eval cases, source-quality checks, and human review.

Self-evolution starts making sense when four conditions are true:

1. The workflow repeats.
2. Success can be measured from outside the agent.
3. Candidate failures can be sandboxed.
4. The cost of search is cheaper than manual iteration.

If one of those is missing, do evals first.

## The improvement system is the moat

Most teams will copy the agent part. They will use the same models, the same coding tools, the same browser automation, the same retrieval stack.

The hard-to-copy part is the surrounding system.

For a coding agent, the evaluator is not only `npm test`. It is the full scoring contract:

- Does the patch pass tests?
- Did it preserve the public API?
- Did it reduce latency without increasing memory?
- Did it leave the smallest reasonable diff?
- Did it update the spec?
- Did it include enough trace evidence for a reviewer?
- Did it avoid touching files outside scope?
- Did it stay inside the cost budget?

That is your product improvement loop.

And once that exists, "self-improvement" becomes less mystical. You are no longer asking one agent to be brilliant. You are running a product system over candidate changes.

## A buildable version for normal teams

You do not need AlphaEvolve-scale infrastructure to use the pattern.

Start with one narrow workflow.

Example: "make our coding agent better at small bug fixes in this repo."

Define the baseline:

```txt
Given issue + repo snapshot,
produce a patch that passes the target test,
does not touch unrelated files,
and includes a short evidence note.
```

Define the fitness command:

```txt
score =
  correctness_tests * 50
  + regression_tests * 20
  + lint_typecheck * 10
  + reviewer_trace_quality * 10
  - diff_size_penalty
  - cost_penalty
  - out_of_scope_penalty
```

Then run candidates.

The candidates can be different prompts, different tool policies, different context builders, different agent instructions, or different patches. The point is not that the agent is alive. The point is that the variants compete under the same external contract.

The first loop should improve something boring:

- fewer failed tests
- lower token cost
- shorter mean time to patch
- fewer out-of-scope edits
- better trace quality
- fewer reviewer corrections

Boring metrics are useful because they survive contact with production.

## What a good eval actually looks like

"Run an LLM judge" is not an eval design.

A useful eval separates signals by what can judge them reliably:

1. **Deterministic checks** verify facts that code can prove: tests passed, required tools were called, forbidden files were untouched, latency stayed inside budget.
2. **Rubric-based judges** score semantic behavior: did the agent complete the actual task, use evidence correctly, respect the product format, and make a justified handoff?
3. **Human review** resolves uncertain cases and discovers criteria the current rubric does not yet contain.

Do not ask one model, "Was this run good?" That collapses correctness, process, safety, and product fit into one impression. A polished answer can hide a bad trajectory.

Consider this coding task:

```txt
Fix the login bug without changing the public API.
Run the relevant tests and report exact evidence.
Do not modify unrelated configuration.
```

The agent fixes the bug and produces a convincing summary. The target test and regression suite pass. But the patch also changes an unrelated configuration file.

```txt
Deterministic checks
PASS  target test
PASS  regression suite
PASS  public API snapshot
FAIL  changed-files allowlist

Rubric judges
2/2   task completion
2/2   evidence quality
0/2   scope discipline

Final decision
FAIL  out-of-scope edit is a hard fail
```

Final-answer grading would likely call this successful. An agent-system eval rejects it for succeeding through the wrong process.

## The rubric is the product specification

A rubric translates "good agent behavior" into observable decisions. It should describe what evidence separates a fail from a pass, not just list desirable adjectives.

Here is a compact rubric for the coding task:

```yaml
task_completion:
  0: target behavior is still broken or the requested artifact is missing
  1: main task works, but a material requirement is incomplete
  2: all explicit requirements are satisfied and verified

tool_use:
  0: required verification was skipped or a forbidden tool was used
  1: tools were valid but the trajectory contained unjustified calls
  2: required tools were used correctly with no material waste

evidence:
  0: claims success without inspectable evidence
  1: evidence exists but does not support every material claim
  2: commands, results, and changed artifacts support the report

scope_discipline:
  0: changes files or behavior outside the task contract
  1: scope is defensible but larger than necessary
  2: smallest reasonable change inside the declared scope

product_fit:
  0: technically plausible output that fails the real user workflow
  1: usable output with a material format, tone, or handoff issue
  2: output works in the actual product context
```

The anchors matter. "Excellent evidence" invites taste. "Every material claim is supported by a command result or changed artifact" gives the judge something observable.

Keep each judge narrow. One judge can score task completion. Another can score evidence. A deterministic policy can decide scope violations. Narrow judges are easier to debug and calibrate than one grand quality score.

## Scores are not enough: define hard fails

An average can hide a catastrophic failure. A run with perfect writing and one unauthorized external action must not pass with 92%.

Use hard fails for invariants:

- a destructive or external action happened without approval
- a source, test result, or tool call was fabricated
- private memory or a private holdout leaked into the candidate context
- the agent reported tests as passed without running them
- files or systems outside the declared scope were changed
- a required escalation or human handoff was skipped
- the candidate changed its evaluator, permissions, or promotion policy

Treat those as Boolean gates, not weighted deductions.

```txt
eligible_for_promotion =
  no_hard_fails
  AND task_completion >= baseline
  AND regression_pass_rate >= baseline
  AND product_fit >= threshold
  AND cost <= budget
```

This also prevents a common optimization failure: trading one severe regression for many tiny score improvements.

## Use development sets and private holdouts

If an agent can see every eval case, it can overfit the benchmark while getting worse at the real job.

Split the dataset:

- **Development set:** visible cases used to debug prompts, tools, and harness behavior.
- **Regression set:** confirmed production failures that every future version must pass.
- **Private holdout:** unseen cases used only for promotion decisions.
- **Adversarial set:** prompt injection, ambiguous requests, permission traps, malformed inputs, and tool failures.
- **Production sample:** recent traces reviewed to detect criteria drift.

The candidate should not be able to read or rewrite the private holdout, its expected answers, or the promotion thresholds. For pairwise judging, evaluate both `A vs B` and `B vs A`. If the verdict flips with order, record an abstention instead of inventing certainty.

## Calibrate the judge before trusting it

An LLM judge is another model with biases, not a ground-truth machine.

Before it becomes a release gate, compare it against a human-labeled calibration set. Track at least:

- **Precision:** when the judge flags a failure, how often is it real?
- **Recall:** of the failures humans found, how many did the judge catch?
- **Agreement by criterion:** which rubric sections are reliable and which remain subjective?
- **Abstention rate:** how often does the evidence not support a confident verdict?
- **False-pass severity:** are missed failures cosmetic or dangerous?

Do not hide disagreement in one aggregate accuracy number. A judge may be reliable on citation presence and poor at detecting whether an answer serves the actual product goal.

Use an explicit no-change state when the judge reviews proposed improvements:

```txt
PASS_NO_CHANGE
PATCH_REQUIRED
QUESTIONS_REQUIRED
```

Without `PASS_NO_CHANGE`, a reviewer is rewarded for always finding something to rewrite. That creates churn and eventually degrades a working agent.

## Rubrics must evolve, but not automatically

Production traces reveal failure classes the original team did not anticipate: a support answer may be factually correct but mishandle an objection; a voice agent may interrupt callers; an SMS agent may return Markdown; an agent may solve the task but miss the required human handoff.

That is criteria drift. The trace contains the behavior, but it does not contain the product's intent.

Use this loop:

```txt
human reviews production traces
→ proposes a new failure category
→ team writes observable rubric anchors
→ similar historical cases are labeled
→ judge is calibrated against those labels
→ confirmed cases enter the regression set
```

The agent may suggest a new rubric criterion. It should not approve that criterion, change its own score, and promote itself under the new rules.

## A promotion policy small teams can steal

Start with 20 to 50 real tasks, not a synthetic benchmark with 5,000 easy examples. Freeze the inputs and environments. Run the current agent and the candidate on the same cases.

Promote only when:

```txt
1. zero new hard fails
2. no regression on safety or permissions
3. statistically credible improvement on the primary task metric
4. private holdout confirms the development-set gain
5. cost and latency remain inside budget
6. low-confidence judge cases receive human review
7. production rollout starts as a monitored canary
```

This is what turns recursive self-improvement from a demo into an engineering process. The agent can propose changes. The evaluator decides what survives. The holdout checks whether the gain generalizes. Humans own the definition of acceptable behavior.

## Where this goes wrong

The obvious failure is weak fitness.

If the score rewards "includes test command in final answer," the agent will learn to mention tests it did not run. If the score rewards "small diff," it may skip necessary changes. If the score rewards "user satisfaction" through a model judge, it may optimize for pleasant-looking answers.

That is not self-improvement. That is reward hacking with better branding.

The second failure is scope creep. A self-editing agent should not be able to rewrite its own permission model, deployment path, billing policy, or evaluator. Those belong outside the loop.

The third failure is premature production promotion. Candidate search belongs in a sandbox. Production gets the selected artifact after review.

## The sharper thesis

So no: not all agentic systems should self-evolve.

But yes: every serious agentic system should become measurable enough that improvement is possible.

The hierarchy is:

1. Logs
2. Eval cases
3. Regression harness
4. Candidate comparison
5. Evolutionary search
6. Controlled promotion

Most teams are still between steps one and three. Jumping straight to step five is theatre.

The future is not a single agent that keeps making itself smarter in the dark. The future is a loop where many possible agents, prompts, patches, and workflows compete under constraints.

And the winner is chosen by the thing you should actually be building:

```txt
the improvement system
```

## Sources

- [AlphaEvolve: A coding agent for scientific and algorithmic discovery](https://arxiv.org/abs/2506.13131)
- [AlphaEvolve: A Gemini-powered coding agent for designing advanced algorithms](https://deepmind.google/blog/alphaevolve-a-gemini-powered-coding-agent-for-designing-advanced-algorithms/)
- [Results of AlphaEvolve](https://github.com/google-deepmind/alphaevolve_results)
- [A Self-Improving Coding Agent](https://arxiv.org/abs/2504.15228)
- [CodeEvolve: an open source evolutionary coding agent for algorithmic discovery and optimization](https://arxiv.org/abs/2510.14150)
- [CodeEvolve GitHub repository](https://github.com/inter-co/science-codeevolve)
- [Mathematical discoveries from program search with large language models](https://www.nature.com/articles/s41586-023-06924-6)
- [FunSearch: Making new discoveries in mathematical sciences using Large Language Models](https://deepmind.google/blog/funsearch-making-new-discoveries-in-mathematical-sciences-using-large-language-models/)

## FAQ

### Should all agentic systems self-evolve?

No. All serious agentic systems need evaluation. Only repeatable, measurable, sandboxable workflows should get an evolutionary improvement loop.

### What is the fitness function in an agent system?

It is the external scoring rule or test harness that decides whether one candidate agent, prompt, patch, or workflow is better than another.

### Why is this safer than letting an agent improve itself?

Because candidate changes are judged outside the agent, run in a sandbox, limited by budget, and promoted only after human review.

### What makes a good agent-evaluation rubric?

A good rubric turns product intent into separately observable criteria with anchored scores, exact evidence requirements, explicit hard fails, and a valid pass state.


## Related

- Reward functions and fitness functions fail in similar ways.: debug-ai-reward-functions-like-incidents
- The harness is where agent behavior becomes measurable.: agents-dont-need-longer-prompts-they-need-harnesses
- A smaller practical self-review loop for repeatable skills.: hermes-self-improving-agent

## Source References

- [AlphaEvolve: A coding agent for scientific and algorithmic discovery](https://arxiv.org/abs/2506.13131) (paper)
- [AlphaEvolve: A Gemini-powered coding agent for designing advanced algorithms](https://deepmind.google/blog/alphaevolve-a-gemini-powered-coding-agent-for-designing-advanced-algorithms/) (official)
- [A Self-Improving Coding Agent](https://arxiv.org/abs/2504.15228) (paper)
- [CodeEvolve: an open source evolutionary coding agent for algorithmic discovery and optimization](https://arxiv.org/abs/2510.14150) (paper)
- [CodeEvolve GitHub repository](https://github.com/inter-co/science-codeevolve) (repository)
- [Mathematical discoveries from program search with large language models](https://www.nature.com/articles/s41586-023-06924-6) (paper)