VII · Verification & ReflectionEmerging★

Frozen Rubric Reflection

also known as Scoped Self-Review, Closed-Set Critic

Constrain reflection to a fixed, hand-authored rubric of criteria so the reviewer cannot invent new ones each run.

This pattern helps complete certain larger patterns —

  • specialisesReflection★★— Have the model review its own output and produce a revised version in one or more passes.
  • used-byDeterministic-LLM Sandwich★— Bracket every LLM call with deterministic checks on both sides.
  • used-byPlanner-Generator-Evaluator Harness·— Decompose a long-running job into three role-isolated agents — a Planner emitting a feature list, a Generator working one chunk per fresh context, and an Evaluator grading against a rubric without seeing the Generator's trace.
  • used-by[evaluation-driven-development]
  • used-byPrompt Variant Evaluation★★— Author multiple variants of the same prompt node, run them as a batch against a shared dataset, and let an automated evaluation flow score them so the winning variant is selected by measurement.

Context

A team uses a model to review the output of another model (or its own previous draft) as a quality gate before shipping. The review needs to be consistent across runs and across users so that two outputs from the same kind of task get judged against the same criteria. Auditors or downstream consumers want to know which checks were performed on each output.

Problem

When the reviewer is given a free-form instruction like 'review this output and flag any issues', it invents fresh criteria on every call: today it notices tone, tomorrow it notices grammar, the day after it notices factual claims. Reviews stop being comparable across runs because they were not measuring the same thing. The reviewer also tends to drift over time, gradually narrowing its attention onto whatever issue it last saw and forgetting categories it used to check. The team has no stable answer to the question 'what did the reviewer actually look for on this run?', which makes the reviewer useless for audit and unreliable as a gate.

Forces

  • Authoring a good rubric is non-trivial up-front work.
  • Rubric drift over time is a separate problem from per-call drift.
  • Some defects fall outside the rubric and go unflagged.

Example

A code-review agent reviews every pull request against a fixed checklist: tests added? naming consistent? error handling? security risk? Without the fixed list, the reviewer would invent new criteria each call and reviews would not be comparable across PRs. Reviewers can only score against the listed criteria — they cannot make new ones up mid-review.

Diagram

Solution

Therefore:

A fixed rubric file (or schema) lists exactly the categories the reviewer may flag. The reviewer prompt includes the rubric and a JSON Schema enforcing it. Temperature is zero. Output validates against the schema; new finding categories are rejected.

What this pattern forbids. The reviewer cannot output finding categories outside the rubric; the JSON schema rejects them.

The smaller patterns that complete this one —

And the patterns that stand alongside it, or against it —

  • composes-withDeterministic-LLM Sandwich★— Bracket every LLM call with deterministic checks on both sides.
  • complementsDream Consolidation Cycle★— Run a deeper, slower reflection pass distinct from per-tick reflection — reading hours of recent thoughts, promoting themes, releasing affective residue, and clearing working memory — so the agent does not accumulate residue indefinitely.
  • complementsBlind Grader with Isolated Context★— Run an evaluator in a separately-allocated context window with access only to the artifact and the rubric, never the producing agent's reasoning trace, so the grader cannot be primed by the producer's framing.
  • complementsSocratic Questioning Agent★— Drive the agent toward its goal by asking the user a sequence of strategic, open-ended questions that surface the user's own latent knowledge, goal, or context — rather than producing an answer directly.
  • complementsCross-Reflection★— Reflection step performed by a *different* agent or foundation model from the original generator, so critique error is decorrelated from generation error.
  • complementsGenerator-Critic Separation★— Strict role separation between a Generator agent that produces drafts and a Critic agent that judges them against pre-defined criteria; the Critic never generates.
  • complementsHuman Reflection★— Reflection loop that explicitly collects human feedback (not approval) on agent plans to improve them, distinct from approval gates where the human only says yes/no.
  • complementsDimensional Synthetic Eval Set★— Generate evaluation inputs not by free-form LLM prompting (which mode-collapses) but by enumerating tuples over explicitly named dimensions and seeding generation from each tuple.

Neighbourhood

Click any neighbour to follow the language. Scroll to zoom, drag to pan.

Used in recipes

References

Provenance