Oversight as Legitimation

Problem

The designated reviewer usually cannot perform oversight. They see only what the system produced and never the counterfactual, so agreeing with it and being unable to disagree with it look identical from outside. The review window is sized by case volume rather than by how long verification takes, and catching a wrong output often means redoing the work the system was bought to save. Spot-checking the outputs that look shaky does not help either, because a model's own confidence does not track when it is wrong. The reviewer frequently has no authority to stop a case and no incentive to dissent. Ben Green's survey of 41 government oversight policies found that people are unable to perform the desired oversight functions and that the policies therefore legitimise faulty and controversial systems without addressing the underlying problems. The sign-off is what gets the system through the gate; its ineffectiveness stays invisible precisely because the mandate was met.

Solution

The anti-pattern is enacted by reading the oversight requirement as a staffing question. A person is designated, the assignment is documented, and the deployment proceeds on that basis; the reviewer is then handed a caseload, a screen showing only the system's output, no protected way to refuse, and a vendor claim that the worst failure mode has been handled. The signature is doing legitimation work rather than control work, and the deploying body, the vendor and the regulator all point to it when an outcome is challenged. The remedy starts by separating the two: measure what fraction of known-wrong outputs the designated reviewer actually catches, using seeded errors or a held-out set where ground truth exists, and treat that number rather than the job title as the evidence that oversight exists. Give the reviewer the counterfactual, the time verification really takes, and a recorded, protected route to refuse. Where the measured catch rate is low or unmeasurable, move the control off the individual: condition procurement on published error rates, require pre-deployment testing on the deploying body's own cases, audit override rates and outcomes, and accept that some systems are then not deployable. Green's proposal is the same move — institutional oversight as the regulating mechanism, with the individual reviewer as one instrument inside it rather than the whole safeguard.

When to use

• Recognising this failure when a legal or policy oversight requirement is met by designating a reviewer and documenting the assignment.
• Reviewing a deployment where the reviewer sees only the system's output, has a review budget set by case volume, or has no recorded way to refuse a case.
• Diagnosing why a sign-off step exists in the process and yet no error has ever been caught by it.
• Assessing a vendor claim that a failure mode has been eliminated, where the designated overseer bears personal responsibility for the result.