Your AI Agent Is Not Reflecting. It Is Defending Its First Answer

The first answer is wrong.

So you ask the agent to reflect.

It rereads its reasoning, adds three paragraphs, sounds more careful — and reaches the same wrong answer with greater confidence.

This is one of the uncomfortable limits of AI self-review: more thinking is not necessarily new thinking.

If the same model starts with the same assumptions, sees the same context, and evaluates its own reasoning, the second pass may not challenge the first. It may rationalize it.

Researchers Tian Liang and colleagues gave this failure a useful name: Degeneration of Thought. Once a model has established confidence in an answer, it can become unable to generate a genuinely different hypothesis through self-reflection — even when its initial position is wrong.

Their proposed remedy is Multi-Agent Debate: one agent proposes, another attacks the proposal, and a judge decides whether the debate has produced a satisfactory solution.

That is more than a prompting trick. It exposes a design choice every agent team needs to make:

Do you want another pass over the same thought,
or do you want a system that can produce a competing thought?

26%

GPT-3.5 baseline accuracy on counter-intuitive arithmetic

27.5%

accuracy after self-reflection

37%

accuracy with multi-agent debate

The circle problem that exposes the failure

The paper uses a deliberately deceptive question:

A small circle with radius r rolls around the outside of a fixed circle
with radius 3r. How many times does the small circle rotate during one trip?

The intuitive answer is three: the large circumference is three times the small one.

It is also wrong. The small circle completes an additional rotation because its center travels around the large circle. The correct answer is four.

In the paper’s example, chain-of-thought produces three. Self-reflection then checks the formula, validates the calculation, and confidently preserves three. The reflection is coherent. Its frame is incomplete.

The opposing debater introduces the missing hypothesis: the calculation must include both rotation around the small circle’s own center and its movement around the large circle. The judge then selects four.

The important move is not “thinking longer.” It is creating a second causal model of the problem.

What changes between reflection and debate

Self-reflection

• One agent owns the answer and the critique.
• The initial frame remains the default.
• Confidence can increase without new evidence.
• The output becomes more polished.

Structured challenge

• A challenger is rewarded for finding a competing explanation.
• The initial frame must survive an explicit attack.
• Disagreement makes hidden assumptions visible.
• The hypothesis space can become wider.

Why self-reflection gets stuck

The paper identifies three contributing factors.

First, bias and distorted perception. The initial answer is often produced by a familiar pattern. Once that pattern is active, the review uses it as the lens through which all later evidence is interpreted.

Second, rigidity. A reflection prompt usually asks the model to inspect its own work. It does not necessarily reward abandoning the work and starting from an incompatible premise.

Third, limited external feedback. Nothing new has entered the system. The agent has the same model, the same task, the same context, and its own previous answer anchoring the next generation.

This explains why “take another look” is a weak correction strategy for failures caused by a hidden assumption. It may work for arithmetic slips, missing fields, inconsistent wording, or facts recoverable from the existing context. It is much weaker when the model needs a different interpretation.

The practical distinction is:

Inspection failure → reflection may help.
Framing failure    → introduce an independent challenge.
Knowledge failure  → retrieve evidence or use a tool.

Calling all three “reasoning problems” leads to the wrong architecture.

A debate is useful only if disagreement is real

Two agent labels do not automatically create two independent thinkers.

If both roles use the same model, identical context, and nearly identical prompts, they may reproduce the same assumptions. “You are Agent B” is not a diversity mechanism.

A useful challenger needs a different objective and, where possible, a different information path:

proposer:
  objective: produce the strongest supported solution
  context: task, available evidence, product constraints

challenger:
  objective: find the first unsupported assumption or construct a counterexample
  context: task, proposal, verification tools

judge:
  objective: decide criterion by criterion, with abstention allowed
  context: task, claims, evidence, challenge — not agent identities

For a coding agent, the challenger should not merely say that the patch “could be improved.” Give it concrete attack surfaces:

• Find an input that breaks the proposed fix.
• Identify a changed behavior outside the issue scope.
• Check whether the test would pass before the patch.
• Search for a public API or configuration contract the patch violates.
• Name the strongest alternative root cause.

For a research agent:

• Find the central claim with the weakest source support.
• Construct the strongest rival explanation.
• Check whether the cited source actually entails the claim.
• Distinguish an observed fact from the author’s inference.

The challenger should be adversarial toward the claim, not toward the proposer. That difference matters.

More disagreement is not always better

The paper found that debate worked best at a modest level of “tit for tat.” Explicit disagreement increased diversity and improved results. Forcing the agents to disagree with every point did not.

At the extreme, the task changes. The agents optimize for winning rather than finding the truth. They preserve incompatible positions, repeat arguments, and turn uncertainty into polarization.

This is a warning for agent prompts such as:

You must find a flaw.
Never agree with the other agent.
Argue until one side wins.

Those instructions guarantee visible conflict, not useful conflict.

The better contract is:

Try to falsify the proposal.
If a claim survives the strongest available counterexample, mark it supported.
If the evidence is insufficient, abstain.
Do not manufacture disagreement after the material issues are resolved.

Engineer productive disagreement

Do

• ✓ assign asymmetric roles with different success conditions
• ✓ require counterexamples, tool evidence, or named assumptions
• ✓ allow agreement after the challenge is resolved
• ✓ stop when the verdict is supported or the budget is exhausted

Avoid

• × clone the same prompt into several agent names
• × reward the longest debate
• × force opposition on every point
• × treat consensus as proof of correctness

The judge becomes the new bottleneck

Multi-agent debate does not remove the evaluator problem. It makes it more visible.

The judge in the paper had two jobs. After each round, it decided whether a correct solution had been found. If the debate hit its iteration limit, it extracted a final answer from the full history.

That creates at least four failure modes:

1. The judge stops too early because one answer sounds complete.
2. It continues after the useful disagreement has been resolved.
3. It selects the more persuasive argument rather than the better-supported one.
4. It loses important evidence inside an increasingly long transcript.

The authors also found a more direct bias: when debaters used different backbone models, the judge tended to prefer the debater built on the same model as itself.

That result should kill the naive idea of an impartial LLM referee. A judge is another model with preferences, positional effects, verbosity bias, and model-family affinity.

Do not ask it which agent “won.” Ask narrow questions:

claim_correctness:
  verdict: supported | contradicted | insufficient_evidence
  evidence: exact test, source, calculation, or counterexample

constraint_compliance:
  verdict: pass | fail
  evidence: exact requirement and observed behavior

material_challenge_resolved:
  verdict: yes | no
  evidence: response to the strongest counterargument

Hide agent identities where possible. Run pairwise comparisons in both answer orders. If A vs B and B vs A produce different winners, abstain or send the case to human review.

Most importantly, keep deterministic evidence outside the debate. A test runner should decide whether tests passed. A schema validator should decide whether the output is valid. A policy gate should decide whether a forbidden tool was used. Do not make the judge infer facts that software can verify.

Two agents were better than four

It is tempting to read a positive multi-agent result and add more agents.

The paper tested that too. Translation quality fell as the number of debaters increased from two to three and four. The authors’ manual analysis pointed to context complexity: debaters forgot earlier views, while the judge struggled to extract the important information from the longer transcript.

More agents create more tokens, more duplicated arguments, more coordination cost, and a harder synthesis problem. They increase potential diversity, but also increase noise.

For a production workflow, start with the smallest topology that can introduce a competing hypothesis:

Bounded challenge loop

one proposal, one real attack, one evidence-based decision

1. 01
   propose
2. 02
   extract claims
3. 03
   challenge strongest assumption
4. 04
   run verification
5. 05
   judge by criterion
6. 06
   accept, revise, or abstain

One proposer and one challenger are enough for the first version. Add specialized roles only when eval traces show a stable failure class that the current pair misses.

A buildable pattern for coding agents

Suppose an agent proposes a fix for an intermittent login bug.

The weak loop is:

Agent: writes patch
Agent: reviews its own patch
Agent: says the patch looks correct

The stronger loop is:

1. Proposer writes the smallest patch and states its root-cause hypothesis.
2. Challenger tries to falsify that hypothesis with one counterexample.
3. Test harness runs target, regression, and adversarial cases.
4. Judge scores only unresolved semantic criteria.
5. System accepts, requests one revision, or escalates.

Use a strict round budget. Most tasks need one challenge and at most one revision. If the same objection survives two rounds, another debate round is unlikely to create new evidence.

stop_when:
  - deterministic_checks_fail
  - all_material_claims_are_supported
  - judge_returns_insufficient_evidence
  - one_revision_is_exhausted
  - cost_or_time_budget_is_reached

This turns debate from theatre into a verification stage.

The production rule

• Use reflection for inspectable omissions and local mistakes.
• Use a challenger when the first answer may rest on a hidden assumption.
• Use tools or retrieval when the system lacks evidence.
• Use a calibrated judge only for decisions deterministic checks cannot make.
• Use an external eval to prove that the whole loop beats the simpler baseline.

What the paper does not prove

The result is useful, but it should not be stretched beyond its evidence.

The main experiments use GPT-3.5-Turbo, GPT-4-0314, and Vicuna models. The tasks are commonsense Chinese-to-English translation and a 200-question counter-intuitive arithmetic dataset. These are deliberately chosen tasks where a plausible first interpretation is often wrong.

That makes them a good test for anchoring. It does not prove that debate improves every agent workflow, every current model, or long-running autonomous systems.

The gains also come with extra model calls and a more complicated failure surface. On the arithmetic set, debate improved GPT-3.5 substantially over its baseline, but it still remained below GPT-4. A stronger model with a verifier may be cheaper and more accurate than several weaker agents debating.

So the right baseline is not “one call versus an impressive multi-agent architecture.” Compare:

• one stronger model
• one model with retrieval or tools
• self-reflection
• best-of-N with deterministic reranking
• proposer plus challenger plus judge

Measure accuracy, hard failures, latency, cost, and human correction rate on the same frozen tasks.

If debate does not win that eval, remove it.

The real lesson

The interesting finding is not that AI agents should argue like humans.

It is that iterative reasoning needs a source of novelty.

Self-reflection reuses the same model, context, and initial anchor. It can catch mistakes, but it can also produce a sophisticated defense of the first answer. Multi-agent debate helps when it introduces a real counter-hypothesis. Then the judge creates a new risk: the system may select familiarity, rhetoric, or model-family affinity instead of truth.

The robust architecture separates the jobs:

Generator → creates a candidate.
Challenger → tries to falsify its central assumption.
Tools      → produce external evidence.
Judge      → resolves narrow semantic criteria.
Eval       → proves whether the entire loop is worth keeping.

Do not add agents because a diagram with five boxes looks advanced.

Add one independent challenge where your failures show that the first answer is anchoring the rest of the run. Give it evidence. Give it a stop rule. Test the judge for bias.

The goal is not more debate.

The goal is a system capable of changing its mind for the right reason.

Sources

• Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate
• Improving Factuality and Reasoning in Language Models through Multiagent Debate
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena