Agent Output Alert Fatigue

Problem

When an agent floods reviewers with findings that are mostly low-value, the humans adapt by disengaging: first they skim, then they approve on autopilot, then they mute the agent entirely. The human-in-the-loop control that justified deploying the agent quietly evaporates, and the agent's genuinely correct findings are now discarded along with the noise. The damage is asymmetric — trust erodes faster from noise than from the occasional missed issue — and near-irreversible, because re-earning a reviewer's attention after they have learned to ignore the agent is far harder than losing it. A volume metric like comments-per-review actively rewards the behaviour that breaks the control.

Solution

Gate the agent's output on confidence so it raises fewer, higher-precision findings; measure usefulness-per-finding, not findings-per-review. A documented post-mortem cut a review agent from 8.3 comments per pull request at 35 percent usefulness to 4.1 at 72 percent and restored reviewer engagement, accepting higher cost per review. Monitor reviewer engagement (resolve rate, mute rate, time-to-skim) as a first-class signal that the oversight control is decaying. Mitigation patterns: cross-encoder reranking or a verifier stage to filter low-value findings before they reach a human; confidence thresholds tuned to the asymmetry. Treat a rising comment count at flat usefulness as an alarm, not progress.

When to use

• Reviewing an assistive agent whose output is high-volume and gated by a human.
• Reviewer engagement with the agent is falling — rising mute or auto-approve rates.
• The agent is measured by output volume rather than usefulness per finding.