Checklist Partial-Credit Scoring

Problem

On long-horizon tasks most runs fail somewhere in the middle, so the bit is nearly always the same bit and the comparison has very little power. A change that carries the agent three steps further on forty tasks and stalls it two steps earlier on thirty others moves the pass rate by roughly nothing, and the report concludes that the change did nothing. Measurement bears this out: across 879 matched task pairs whose binary outcome did not change, 20.9 percent improved by more than 0.10 of graded progress while 18.7 percent regressed by the same margin, and a pass-rate harness discards all of it. The team is left tuning against a signal that only registers when a task crosses the finish line.

Solution

Each task is decomposed at authoring time into an ordered list of checkpoints naming the intermediate states a solution has to pass through. Every checkpoint carries a check: a scripted assertion against the final environment or the recorded trace wherever the check can be made deterministic, and a short rubric graded by a judge model only where it cannot. A run is scored as the weighted fraction of checkpoints whose check passed. The ordering is a gate — credit for a later checkpoint requires the earlier ones — so a run cannot harvest scattered intermediate states it never built on. One checkpoint is reserved for comparing the final report against the trace: a substantive claim with no matching entry in the recorded trace costs the run credit instead of earning it. The official binary verdict is computed exactly as before and reported alongside the graded score, so the auditable number survives and the two can disagree in the open. Version comparisons are then made on matched pairs of runs over the same tasks, which is where the graded score has power the bit does not: it separates the pairs that improved from the pairs that regressed instead of averaging them into a flat delta.

When to use

• Most runs on the suite fail partway through, so the pass rate barely moves between agent versions and comparisons have little power.
• Tasks are long-horizon and pass through identifiable intermediate states that can be checked from the environment or the recorded trace.
• The comparison is between versions of the same agent over the same tasks, where matched-pair scoring separates improvement from regression.
• Knowing where a run stopped matters as much as knowing that it stopped.