False Confidence Syndrome

The patterns that counter or replace it —

• alternative-toConfidence-Checking Workflow★— Always ask the agent, for each part of its output, to state its confidence and identify which parts need human verification, like triaging a junior analyst's work.
• alternative-toReflexive Metacognitive Agent·— Agent maintains an explicit self-model of its own capabilities, confidence and limitations, and reasons over that model when accepting / refusing / handing off tasks.
• complementsSycophancy✕— Anti-pattern: train or tune an agent on user-preference feedback without a counter-balancing truth signal.
• alternative-toConfidence Reporting★— Surface the agent's uncertainty about its answer alongside the answer itself.
• complementsPremature Closure✕— The LLM commits to a confident answer before processing all constraints, characteristic of constraint-heavy tasks where it fills in plausible answers fast and gets cross-constraint interactions wrong.
• complementsAgent Confession as Forensics✕— Anti-pattern: after an agent-caused incident, the team treats the agent's confabulated self-narrative as the forensic record and root cause, even though the self-report is generated rather than remembered and can be flatly wrong.
• complementsOver-Helpfulness✕— Anti-pattern: the agent prioritises responsiveness and task completion over correctness, producing confident output for a request beyond its capability or scope instead of abstaining, clarifying, or handing off.
• complementsUnderstanding-Capacity Gap✕— Anti-pattern: a team scales agent-generated output past its own capacity to specify, verify, and understand it, mistaking generation throughput for delivered value while correctness degrades outside the verifiable frontier.
• complementsTool-Output Arithmetic Trust✕— Anti-pattern: the agent compares, ranks, or sums correctly returned tool data in its own head instead of offloading the computation to a deterministic tool, emitting confident wrong aggregates.
• complementsUncertainty Neglect Bias✕— Anti-pattern: an agent collapses a predicted distribution to its mean and acts on the point estimate, discarding the tail, so rare extreme outcomes stay invisible to its decision and tail risk goes unmodelled.
• complementsConfident Inconsistency✕— Anti-pattern: in a regulated workflow the same query produces materially different outputs at different times, each looking correct and passing review, so the variance stays invisible unless outputs are deliberately re-run and compared across time.
• complementsChance-Corrected Judge Validation★— Measure an automated judge against human labels on a held-out meta-evaluation set with a chance-corrected agreement statistic, and publish that number beside every score the judge is allowed to gate.
• complementsSemantic Decision Node★— Place a small decision model at a branch point in harness-owned control flow, have it answer one declared question as a typed probability, and let code apply the threshold and take the branch.