Chance-Corrected Judge Validation

also known as Judge Meta-Evaluation, Kappa-Gated Judge, Judge Agreement Licence

Measure an automated judge against human labels on a held-out meta-evaluation set with a chance-corrected agreement statistic, and publish that number beside every score the judge is allowed to gate.

Context

Open-ended outputs and agent trajectories have no exact-match answer key, so a model-based judge scores them instead of a human reviewer. Those scores then start deciding things: which release ships, which rollout continues, which model sits where on a leaderboard, which outputs skip review. The judge's own quality is usually asserted from a single figure, the share of cases where the judge and a human reviewer chose the same label on a sample.

Problem

That share counts every case where judge and human happened to land on the same label, including the ones a rater guessing the majority label would also have got. On a skewed label distribution it stays high while the judge discriminates almost nothing. A systematic study of 21 judges across roughly 541,000 judgements found the gap between exact-match agreement and Cohen's kappa runs 33 to 41 percentage points on MT-Bench, and that judge rankings move by up to 14 positions once the correction is applied. A number that flattering is not a licence to gate anything, yet it is routinely used as one.

Forces

Automated judging scales to volumes human annotation cannot reach; one study covered roughly 541,000 judgements, a scale no annotation panel would sustain.
Exact-match agreement is the cheapest number to compute and the one that overstates the judge most, by 33 to 41 percentage points on MT-Bench.
A human-labelled meta-evaluation set costs annotator time and has to establish its own reliability first; the Counsel agentic-task dataset reports a Krippendorff's alpha of 0.78 across annotators before any judge is scored against it.
Agreement measured on one task distribution does not carry to another: the same judges that look adequate on final answers reach only about 0.65 AUROC on tau2-bench trajectories and 0.54 on AppWorld API-call traces.
Every judge-model upgrade or rubric edit changes the thing that was measured, so a validated number decays quietly while the scores it licensed keep flowing.

When to use

A judge's scores decide something: a release gate, a rollout, a leaderboard position, or which outputs skip human review.
Human labels can be collected for a held-out sample drawn from the same task distribution the judge will score.
The label distribution is skewed enough that raw agreement can look high while discrimination is close to chance.
Judge models, rubrics or task distributions change often enough that a one-off calibration goes stale unnoticed.

When not to use

The criterion has an exact-match or executable ground truth, in which case the check replaces the judge rather than validating it.
The scores are exploratory and gate nothing, so an annotated meta-evaluation set is not repaid.
Human labels on the task are not yet reliable enough to measure against; inter-annotator reliability has to be established first.
The label distribution is so extreme that a chance-corrected statistic rests on a handful of disagreements and reports noise.

Example

A team ships a support assistant and lets a model judge decide which replies are good enough to send without review. The judge and a human reviewer agree on 88 percent of a sampled week, which sounds convincing until someone notices that 85 percent of replies are fine anyway, so the judge barely beats a rater who says fine every time. Recomputed as Cohen's kappa the judge scores close to zero, and the gate it had been running goes back to advisory until a properly labelled set exists.

Diagram

flowchart TD A["Held-out sample of the target task distribution"] --> B["Human labels + inter-annotator reliability"] A --> C["Judge labels the same items"] B --> D["Chance-corrected agreement: kappa or alpha"] C --> D D --> E{"Meets the threshold for this decision?"} E -- "yes" --> F["Judge may gate; every score carries set version, judge model, rubric version, agreement"] E -- "no" --> G["Judge stays advisory"] H["Judge model, rubric or task distribution changes"] --> A I["Raw exact-match agreement only"] -. "flatters by 33-41 pp" .-> F

Solution

Therefore:

Build a held-out sample of the task distribution the judge will actually score and have people label it, reporting inter-annotator reliability so the labels themselves are known to be stable. Run the judge over the same items and compare its labels to the human ones with a statistic that discounts agreement reachable by chance: Cohen's kappa for two raters over the same items, Krippendorff's alpha where there are more raters or missing labels. Set a threshold per decision, so a judge licensed to flag outputs for review is not automatically licensed to block a release. Record the measured agreement together with the meta-evaluation set version, the judge model and the rubric version, and attach that record to every score the judge emits, so a reader can tell an audited number from an unaudited one. Re-run the measurement whenever any of those three inputs changes, and treat a judge below threshold as advisory rather than as a gate. Raw exact-match agreement may still be reported, but only next to the corrected statistic, never in place of it.

What it gives you

A judge that looks accurate only because the label distribution is skewed is caught before it gates a release.
Scores carry provenance: meta-evaluation set version, judge model, rubric version and measured agreement travel with the number.
Comparisons between judges become stable, rather than reordering by up to 14 positions the moment the correction is applied.
The decay of a judge's licence becomes visible, because a model upgrade or rubric edit invalidates a dated measurement instead of silently outliving it.

What it costs you

The meta-evaluation set is a standing annotation cost and must establish its own reliability, which the Counsel dataset reports as a Krippendorff's alpha of 0.78 rather than assuming.
Chance-corrected statistics are unstable on very skewed label distributions, so a high kappa computed over few disagreements can be as misleading as the raw agreement it replaced.
A single headline agreement number hides slices where the judge cannot discriminate at all, and a judge validated on final answers stays unvalidated on trajectories.
The threshold is a judgement call; no measured agreement level makes a judge safe for every decision it might be pointed at.

What this pattern forbids. A judge may not gate a release, a rollout, a leaderboard position or an escape from human review before its chance-corrected agreement with human labels on a held-out meta-evaluation set has been measured and published; raw exact-match agreement is never sufficient on its own, and the licence lapses when the judge model, the rubric or the task distribution changes.

The smaller patterns that complete this one —

usesEval Harness★★— Run a held-out dataset against agent versions to detect regressions and measure improvement.

And the patterns that stand alongside it, or against it —

complementsLLM-as-Judge★★— Use an LLM to score open-ended outputs against rubric criteria when no exact-match metric applies.
complementsAgent-as-a-Judge★— Evaluate an agent's full trajectory (steps, tool calls, intermediate states) by another agent rather than scoring only the final output.
complementsScorer Live Monitoring★— Score agent outputs asynchronously in production with non-blocking scorers that observe, alert, and log but do not regenerate the output.
complementsSampled Prompt Trace Eval★— Capture full prompt/response/metadata traces from production into a monitoring dataset, but only run LLM-judge evaluation on a random sample so monitoring cost stays bounded as traffic grows.
complementsFrozen Rubric Reflection★— Constrain reflection to a fixed, hand-authored rubric of criteria so the reviewer cannot invent new ones each run.
complementsBlind Grader with Isolated Context★— Run an evaluator in a separately-allocated context window with access only to the artifact and the rubric, never the producing agent's reasoning trace, so the grader cannot be primed by the producer's framing.
complementsEval as Contract★★— Treat the eval suite as the contract the agent must satisfy; releases ship only if evals pass.
complementsFalse Confidence Syndrome✕— Anti-pattern: the model produces incorrect answers with the same high confidence as correct ones, failing to vary its expressed certainty with its actual reliability — Oxford-documented for constraint-heavy prompts.
complementsJudge-Channel Injection✕— Anti-pattern: score a candidate artifact with an LLM judge that reads it as ordinary content, so the party whose score is at stake can write text into the judge's own prompt.
complementsSimulated-User Rollout Evaluation★— Evaluate a conversational agent by pairing it with a second model that plays the user from a persona and a hidden goal, then score the final environment state rather than the transcript.