XIV · Anti-PatternsAnti-pattern✕

Prior-Estimate Anchoring

also known as Anchored Second Opinion, Prior-Score Contamination, Metadata Anchoring

Anti-pattern: pass a prior score, attempt counter, or earlier verdict into the context of the stage meant to judge independently, so the second judgement is pulled toward the first and agreement is manufactured.

Context

A pipeline judges the same artifact more than once. A refinement loop scores a draft, sends it back for revision, and scores it again; a review stage asks for a second opinion on an earlier verdict; a triage agent inherits the previous shift's severity rating; a negotiation agent reads the other side's opening offer. The harness carries its own bookkeeping forward for good reasons — the revision number, the attempt counter, the prior score, the earlier classification are what detect convergence, track progress, and give the next stage background. That state ends up in the same prompt as the artifact under judgement.

Problem

The second judgement is supposed to be independent, and it is not. A prior score included only as context metadata shifts the new rating toward its value: across 185,271 successful evaluations the shift reached a Cohen's d of 0.71, and 48% of the corrections the judge would otherwise have made were blocked, so a draft that scored low stayed low and one that scored high stayed high. The pull is not an artifact of weak models — a fourteen-model benchmark found frontier systems above 95% accuracy on the anchor-free control condition still moved, so control accuracy does not predict resistance to a plausible anchor. It is not confined to numbers either: a categorical prior analysis result shifts a code-vulnerability agent's call at 23.5% cross-model susceptibility, behind framing at 33.2% and ahead of the halo effect at 18.4%. Chain-of-thought does not remove the effect, and neither does an instruction to disregard the metadata. Agreement between stages therefore measures the strength of the pipeline's own bookkeeping rather than the quality of the artifact.

Forces

  • A refinement loop needs its own bookkeeping — revision number, attempt count, previous score — to detect convergence and stop, so the state that anchors the judge is the same state that makes the loop terminate.
  • More background usually improves a judgement, and the earlier verdict is genuinely informative, so removing it from the judging call feels like withholding evidence.
  • The anchor is cheap to pass and invisible in the transcript: the second judgement reads as considered whether or not it moved.
  • Telling the judge to ignore the metadata costs nothing and looks like a fix, but the measured total effect survives both that warning and chain-of-thought prompting.
  • Model quality is the usual lever for judgement problems, yet accuracy on the anchor-free control condition does not predict how far a plausible anchor moves the answer, so upgrading the judge does not settle it.

Example

A code-review loop scores a patch 4 out of 10, the author agent rewrites it, and the same reviewer is asked to score the new version. The rewrite is much better, but the reviewer's prompt still carries the line "revision 2, previous score 4", and the new score comes back as 5. A reviewer handed only the rewritten patch and the rubric scores it 8. The loop's own bookkeeping, not the patch, decided the number.

Diagram

Solution

Therefore:

Separate the loop's bookkeeping from the judging context. The orchestrator retains the revision number, the attempt counter, and every prior score for convergence detection and logging, while the judging call receives only the artifact and the rubric — no field, header, or preamble naming a previous value, and no summary that implies one. Where a second opinion has to follow a first, run both judgements from independent contexts and combine them afterwards instead of nesting one inside the other. When prior state genuinely has to stay visible, blind or randomise it so it cannot act as a numeric target; the 6G control work replaces fixed heuristic anchors with a randomised draw for exactly this reason. Measure what is left with a paired experiment that re-scores the same artifacts with and without the metadata, and treat a disregard instruction as a note rather than a control, because the measured effect persists through it.

What this pattern forbids. A stage whose judgement is meant to be independent must not receive a prior score, attempt counter, or earlier verdict in its context, not even as metadata paired with an instruction to disregard it; the loop's bookkeeping stays with the orchestrator, and agreement between stages cannot be counted as confirmation unless the later stage was blind to the earlier one.

The patterns that counter or replace it —

  • alternative-toBlind Grader with Isolated Context★— Run an evaluator in a separately-allocated context window with access only to the artifact and the rubric, never the producing agent's reasoning trace, so the grader cannot be primed by the producer's framing.
  • complementsLLM-as-Judge★★— Use an LLM to score open-ended outputs against rubric criteria when no exact-match metric applies.
  • complementsEvaluator-Optimizer★★— One LLM generates; another evaluates and feeds back; loop until criteria are met.
  • complementsUncertainty Neglect Bias✕— Anti-pattern: an agent collapses a predicted distribution to its mean and acts on the point estimate, discarding the tail, so rare extreme outcomes stay invisible to its decision and tail risk goes unmodelled.
  • complementsSycophancy✕— Anti-pattern: train or tune an agent on user-preference feedback without a counter-balancing truth signal.
  • complementsLost in the Middle (Positional Bias)✕— LLM accuracy on retrieving information from long contexts drops sharply when relevant content sits in the middle of the prompt rather than at the start or end.
  • complementsMemo-As-Source Confusion✕— Anti-pattern: the agent cites its own past memos as ground truth instead of re-verifying them against the artifacts they describe, accumulating false confidence in stale summaries.
  • complementsConsensus-Averaging Over Expertise✕— Anti-pattern: a self-organising LLM team pursues integrative compromise, averaging expert and non-expert views instead of weighting the known expert, so team output falls below the best member and degrades further as the team grows.
  • complementsHeterogeneous-Model Council with Synthesis Judge★— Three or more role-specialized personas run on different model architectures in parallel; a synthesis judge — given only their structured JSON, not the original input — produces the final verdict.
  • complementsSelf-Consistency★★— Sample the same question multiple times at non-zero temperature and aggregate by majority or judge to mitigate hallucination.
  • complementsAI-Targeted Comment Injection✕— Anti-pattern: an attacker seeds source files with thousands of lines of repetitive natural-language comments designed to instruct the model code auditors / agents that may read the file — not to communicate with human developers.

Neighbourhood

Click any neighbour to follow the language. Scroll to zoom, drag to pan.

References

Provenance