Chance-Corrected Judge Validation

Problem

That share counts every case where judge and human happened to land on the same label, including the ones a rater guessing the majority label would also have got. On a skewed label distribution it stays high while the judge discriminates almost nothing. A systematic study of 21 judges across roughly 541,000 judgements found the gap between exact-match agreement and Cohen's kappa runs 33 to 41 percentage points on MT-Bench, and that judge rankings move by up to 14 positions once the correction is applied. A number that flattering is not a licence to gate anything, yet it is routinely used as one.

Solution

Build a held-out sample of the task distribution the judge will actually score and have people label it, reporting inter-annotator reliability so the labels themselves are known to be stable. Run the judge over the same items and compare its labels to the human ones with a statistic that discounts agreement reachable by chance: Cohen's kappa for two raters over the same items, Krippendorff's alpha where there are more raters or missing labels. Set a threshold per decision, so a judge licensed to flag outputs for review is not automatically licensed to block a release. Record the measured agreement together with the meta-evaluation set version, the judge model and the rubric version, and attach that record to every score the judge emits, so a reader can tell an audited number from an unaudited one. Re-run the measurement whenever any of those three inputs changes, and treat a judge below threshold as advisory rather than as a gate. Raw exact-match agreement may still be reported, but only next to the corrected statistic, never in place of it.

When to use

• A judge's scores decide something: a release gate, a rollout, a leaderboard position, or which outputs skip human review.
• Human labels can be collected for a held-out sample drawn from the same task distribution the judge will score.
• The label distribution is skewed enough that raw agreement can look high while discrimination is close to chance.
• Judge models, rubrics or task distributions change often enough that a one-off calibration goes stale unnoticed.