VII · Verification & ReflectionEmerging★

Modality-Conflict Arbitration

also known as Cross-Modal Arbitration, Cross-Modal Disagreement Detection, Per-Channel Claim Reconciliation

Have each observation channel emit its own recorded claim, treat disagreement between channels as a detected event, and resolve it against a pre-declared per-claim-type modality authority rather than an implicit fused guess.

Context

An agent observes one situation through more than one channel at the same time: a call arrives as both an audio stream and a transcript, a web page arrives as both an accessibility tree and a screenshot, a clip arrives as both a caption and its frames, a clinical intake arrives as text, voice and video together. Each channel carries something the others do not. Standard practice fuses them early — the channels are encoded into one representation, or the cheapest channel is rendered into text and the rest are dropped — and a single generation pass produces one answer. Nothing in that pass records what each channel said on its own.

Problem

Channels do not always agree, and a fused pass has no way to report that they disagreed. The measurements are consistent across modalities. On audio-grounded dialogue, text-only reasoning exceeds 90 percent accuracy when transcript and acoustics agree but falls to 33-48 percent when they conflict, and audio-native models still pick the transcript-biased answer in roughly 30 to 40 percent of conflict cases. Attention-level analysis shows the bias is structural rather than incidental: a distributed set of heads drives generation toward the textual premise while only a small localised set resists it, so the erroneous premise wins by routing. Benchmarks over long audio-visual material find that omni-modal models caption and answer questions on temporally aligned content yet cannot reliably perceive that the two tracks contradict each other. The output is not a hedged answer but a confident one, and the contradiction that was present and detectable in the inputs leaves no trace.

Forces

Early fusion is cheaper and usually more accurate than keeping the channels apart, because most observations are consistent and one representation exploits their overlap.
The textual channel is the cheapest to produce and the one models see most in training, so the prior toward it is strong, systematic and invisible from the output alone.
Disagreement between channels is itself information about the world — sarcasm, a dubbed clip, a mislabelled screenshot, an edited document — so collapsing it removes the only evidence that the situation is unusual.
Which channel deserves to win depends on the kind of claim being made rather than on any fixed ranking, so a single global modality preference is wrong in a large share of the cases where it matters.
Recording a claim per channel costs an extra pass and adds new places to be wrong, and a conflict nobody can resolve leaves the agent with no answer to give.

When to use

One situation is observed through two or more channels that can each support a claim independently, such as transcript and acoustics, screenshot and accessibility tree, or caption and frames.
Disagreement between the channels is itself informative — sarcasm, dubbing, editing, mislabelled interface elements, stale document text.
A wrong answer costs more than a late one, so abstaining or escalating on an unresolvable conflict is acceptable.
Domain experts can state in advance which channel is authoritative for which kind of claim.

When not to use

Only one channel carries the information at issue, so there is nothing to compare a claim against.
The channels are derived from each other, such as an acoustic branch conditioned on the transcript, so agreement is guaranteed by construction and carries no information.
Latency or cost budgets rule out a claim extraction pass per channel and the task tolerates the error rate of the fused answer.
No defensible authority per claim type can be stated, so the resolution would collapse into a fixed global preference for one modality.

Example

A support agent processes a recorded call. The transcript reads "that is exactly what I needed, thanks", so the text branch marks the caller satisfied and closes the ticket. The audio branch heard a flat, clipped delivery and marks the caller annoyed. Because the two claims were recorded separately, the disagreement is caught, the declared authority for tone is the acoustic channel, and the call goes to a human instead of being closed.

Diagram

flowchart TD A["Observation: one situation, several channels"] --> B["Transcript claim"] A --> C["Acoustic claim"] A --> D["Visual claim"] B --> E{"Claims agree?"} C --> E D --> E E -- "yes" --> F["Answer, agreement recorded"] E -- "no" --> G["Conflict recorded as an event"] G --> H{"Authority declared for this claim type?"} H -- "yes" --> I["Apply authoritative channel, keep losing claim"] H -- "no" --> J["Abstain or escalate to a human"]

Solution

Therefore:

Split the observation into per-channel claims before anything is fused. Each channel — transcript, acoustics, frames, accessibility tree, document text — answers the question in its own right and records a typed claim with its own confidence and the span of evidence it rests on. A comparison step then checks the claims slot by slot and marks agreement or disagreement explicitly, so a contradiction becomes a named event instead of a difference that dissolves inside a fused representation. Resolution runs against a table declared in advance and indexed by claim type, not by channel alone: what was literally said is settled by the transcript, how it was said by the acoustics, what was physically present by the frames, what an interface element is called by the accessibility tree. Where the table has an entry, the authoritative channel is applied and the losing claim is kept next to the decision as the reason the answer is uncertain. Where it has no entry, or where the authoritative channel is itself low-confidence, the agent abstains or escalates instead of picking. A reflection step revisits only the disagreeing slots, which keeps the extra reasoning proportional to the conflict rather than to the whole observation.

What it gives you

A contradiction between channels leaves the system as a recorded event rather than as a confident single answer, so a reviewer can see that the observation was in tension.
The uncertainty signal comes from a comparison the agent can actually compute, instead of being asked of the model as an introspective confidence estimate.
The per-claim-type authority table is inspectable, can be argued about by domain experts, and does not shift with prompt wording the way an implicit preference does.
The losing claim is retained beside the decision, so a case can be re-decided later without re-running perception.

What it costs you

Running a claim extractor per channel multiplies inference cost and adds a channel's worth of new error sources, including disagreements that are extraction artefacts rather than real conflicts.
The authority table has to be authored per domain and goes stale; an entry naming the wrong channel converts a detectable conflict into a confidently wrong resolution with an audit trail attached.
Abstention has a price: where the conflict is genuine but a decision is still needed, the agent hands back nothing.
Per-channel claims can be correlated through a shared encoder or through an acoustic branch conditioned on the transcript, so apparent agreement may be one channel counted twice.

What this pattern forbids. An answer may not be emitted from a fused representation while the channels disagree: each channel must record its own claim before fusion, a detected conflict cannot be settled by an implicit or global modality preference, and only the authority declared in advance for that claim type — or abstention and escalation — may resolve it.

And the patterns that stand alongside it, or against it —

complementsMultimodal Guardrails★— Input and output guardrails that operate across modalities (vision, audio, file) rather than text only — handling e.g. malicious instructions embedded in image OCR or audio transcription.
complementsPriority Matrix (Conflict Resolution)★— Pre-define how the agent must resolve specific classes of goal conflicts via a human-authored lookup table — transforming the agent from a decision-maker (where it fails on competing objectives) into a decision-implementer.
alternative-toSelf-Consistency★★— Sample the same question multiple times at non-zero temperature and aggregate by majority or judge to mitigate hallucination.
complementsCross-Reflection★— Reflection step performed by a *different* agent or foundation model from the original generator, so critique error is decorrelated from generation error.
complementsEpistemic Fault-Domain Quorum★— Credit a multi-agent quorum only with the resilience its distinct upstream roots justify, admitting the coalition when the number of independent root faults needed to compromise it clears a declared threshold.
complementsConfidence Reporting★— Surface the agent's uncertainty about its answer alongside the answer itself.
complementsMandatory Red-Flag Escalation★★— Maintain a deterministic set of high-risk triggers so that on any match the agent immediately aborts its workflow and hands off to a human, without weighing whether to escalate.
complementsFoveated Perception Escalation★— Treat observation fidelity as a budget the agent allocates: perceive the whole surface coarsely, let the reasoning name the region that decides the answer, then re-observe only that region at full resolution.

Neighbourhood

Click any neighbour to follow the language. Scroll to zoom, drag to pan.

References

Provenance

Source: patterns/modality-conflict-arbitration.md on GitHub · commit 240ffcd · view history
Added to catalog: 2026-09-09
Last updated: 2026-09-09
Contribute: open an issue or PR at github.com/agentpatternscatalog/patterns.