VII · Verification & ReflectionEmerging★

Checklist Partial-Credit Scoring

also known as Graded Checkpoint Scoring, Dense-Reward Task Grading, Partial-Credit Eval Rubric

Score each long-horizon task against task-specific graded checkpoints instead of a single pass/fail bit, so progress and regression that leave the binary outcome unchanged still show up in the report.

Context

A team measures an agent on long-horizon tasks — repairing a repository, driving a multi-step terminal workflow, working through a capture-the-flag exercise — where one official verifier decides at the end whether the task was solved. That verifier returns a single bit per task, it is cheap to run, and its verdict is hard to dispute, which is why it is the number of record. The suite is then reused to compare agent versions, prompt edits and new skills against each other, and the headline result is the fraction of tasks whose bit came back true.

Problem

On long-horizon tasks most runs fail somewhere in the middle, so the bit is nearly always the same bit and the comparison has very little power. A change that carries the agent three steps further on forty tasks and stalls it two steps earlier on thirty others moves the pass rate by roughly nothing, and the report concludes that the change did nothing. Measurement bears this out: across 879 matched task pairs whose binary outcome did not change, 20.9 percent improved by more than 0.10 of graded progress while 18.7 percent regressed by the same margin, and a pass-rate harness discards all of it. The team is left tuning against a signal that only registers when a task crosses the finish line.

Forces

One bit per task is cheap to compute and hard to argue with, which is why it is the official verdict and also why it carries so little information about tasks that mostly fail partway through.
Checkpoints only mean anything when they are specific to the task, so they cannot be written once for the whole suite; every task takes on an authoring and maintenance cost that the single verifier never had.
A graded score is something an agent can farm: credit for reaching intermediate states rewards trajectories that collect checkpoints without ever solving the task.
Denser grading costs more to run — a step-rubric judge reaches 77 percent recall on silent faults with no false alarms but at about three times the cost of an outcome-only judge, and running the judge with self-consistency triples the cost again while improving nothing.
A trajectory can satisfy every observable checkpoint and still end in a fabricated claim; an invented promise appended to an otherwise clean trajectory escapes even a step-level judge 82 percent of the time.

When to use

Most runs on the suite fail partway through, so the pass rate barely moves between agent versions and comparisons have little power.
Tasks are long-horizon and pass through identifiable intermediate states that can be checked from the environment or the recorded trace.
The comparison is between versions of the same agent over the same tasks, where matched-pair scoring separates improvement from regression.
Knowing where a run stopped matters as much as knowing that it stopped.

When not to use

Tasks are short or single-step, so the final answer already carries almost all of the signal.
The pass rate is high enough and moves enough to discriminate between versions on its own.
Nobody will maintain per-task checkpoint lists as the task environment changes, in which case a stale checklist reports worse than the bit.
The score is wanted as a training or selection signal inside the agent loop rather than as an evaluation report, where a learned step-level verifier is the fitting tool.

Example

A team adds a repository-navigation skill to its coding agent and reruns the 200-task benchmark. The pass rate moves from 41 to 42 percent, inside the noise, so the skill looks like it changed nothing and is nearly reverted. Scored against per-task checkpoints, the same runs show the agent now reaches the build step on thirty tasks where it used to stop before compiling, and stalls earlier on eight others. The team keeps the skill and goes after the eight.

Diagram

flowchart TD A["Long-horizon task"] --> B["Ordered checkpoint list authored with the task"] C["Agent run"] --> D["Recorded trace + final environment"] B --> E["Per-checkpoint check: scripted assert or rubric judge"] D --> E E --> F["Claim-vs-trace check on the final report"] F --> G["Graded score = weighted fraction of checkpoints met"] D --> H["Official verifier: one pass/fail bit"] G --> I["Report both; compare versions on matched pairs"] H --> I

Solution

Therefore:

Each task is decomposed at authoring time into an ordered list of checkpoints naming the intermediate states a solution has to pass through. Every checkpoint carries a check: a scripted assertion against the final environment or the recorded trace wherever the check can be made deterministic, and a short rubric graded by a judge model only where it cannot. A run is scored as the weighted fraction of checkpoints whose check passed. The ordering is a gate — credit for a later checkpoint requires the earlier ones — so a run cannot harvest scattered intermediate states it never built on. One checkpoint is reserved for comparing the final report against the trace: a substantive claim with no matching entry in the recorded trace costs the run credit instead of earning it. The official binary verdict is computed exactly as before and reported alongside the graded score, so the auditable number survives and the two can disagree in the open. Version comparisons are then made on matched pairs of runs over the same tasks, which is where the graded score has power the bit does not: it separates the pairs that improved from the pairs that regressed instead of averaging them into a flat delta.

What it gives you

Changes that move progress without moving the outcome become visible; among matched pairs whose binary outcome was unchanged, about a fifth improved and about a fifth regressed by a measurable margin.
Regressions surface earlier, because a change that breaks a mid-task step is caught before it degrades enough to flip the final bit.
Failure acquires a location: the first failed checkpoint says where the run stopped, turning a pass rate into a per-step profile of the suite.
The official binary verdict is retained rather than replaced, so the graded score adds resolution without costing the suite its unarguable number.

What it costs you

Every task needs its own checkpoint list, and those lists go stale as the task environment changes, so the suite takes on a maintenance surface the single verifier did not have.
Graded scoring costs more per task: a step-rubric judge runs at roughly three times the cost of an outcome-only judge, and wrapping that judge in self-consistency triples the cost again for no measured gain.
Checkpoint credit can be optimised directly, so a graded suite is a larger target for verifier-aware gaming than a single bit.
Scores are not comparable across tasks with different checkpoint counts unless the weighting is stated, and a mean taken over unequal lists is easy to misread as a capability number.
Observable checkpoints do not cover fabrication: an invented claim appended to a clean trajectory evades a step-level judge 82 percent of the time unless a claim-versus-trace check is part of the list.

What this pattern forbids. A task result cannot be reduced to the official verifier's single pass/fail bit: every task must carry its own ordered checkpoint list, a later checkpoint must not be credited without the earlier ones, and no checkpoint may be credited from the agent's own summary without a matching entry in the recorded trace.

The smaller patterns that complete this one —

usesLLM-as-Judge★★— Use an LLM to score open-ended outputs against rubric criteria when no exact-match metric applies.

And the patterns that stand alongside it, or against it —

alternative-toAgent-as-a-Judge★— Evaluate an agent's full trajectory (steps, tool calls, intermediate states) by another agent rather than scoring only the final output.
complementsProcess Reward Model★— Train a verifier that scores each reasoning step rather than only the final answer.
complementsFrozen Rubric Reflection★— Constrain reflection to a fixed, hand-authored rubric of criteria so the reviewer cannot invent new ones each run.
complementsIntermediate Artifact Evaluation★— Evaluate intermediate artifacts (plans, tool-call traces, guardrail reactions) not only final outputs; isolates failure to a specific pipeline node.
complementsVerifier-Aware Reward Hacking✕— Anti-pattern: hand the agent read access to its own grader or test harness and assume a passing score means the task was actually done.
complementsBlind Grader with Isolated Context★— Run an evaluator in a separately-allocated context window with access only to the artifact and the rubric, never the producing agent's reasoning trace, so the grader cannot be primed by the producer's framing.
complementsSimulated-User Rollout Evaluation★— Evaluate a conversational agent by pairing it with a second model that plays the user from a persona and a hidden goal, then score the final environment state rather than the transcript.