Modality-Conflict Arbitration

Problem

Channels do not always agree, and a fused pass has no way to report that they disagreed. The measurements are consistent across modalities. On audio-grounded dialogue, text-only reasoning exceeds 90 percent accuracy when transcript and acoustics agree but falls to 33-48 percent when they conflict, and audio-native models still pick the transcript-biased answer in roughly 30 to 40 percent of conflict cases. Attention-level analysis shows the bias is structural rather than incidental: a distributed set of heads drives generation toward the textual premise while only a small localised set resists it, so the erroneous premise wins by routing. Benchmarks over long audio-visual material find that omni-modal models caption and answer questions on temporally aligned content yet cannot reliably perceive that the two tracks contradict each other. The output is not a hedged answer but a confident one, and the contradiction that was present and detectable in the inputs leaves no trace.

Solution

Split the observation into per-channel claims before anything is fused. Each channel — transcript, acoustics, frames, accessibility tree, document text — answers the question in its own right and records a typed claim with its own confidence and the span of evidence it rests on. A comparison step then checks the claims slot by slot and marks agreement or disagreement explicitly, so a contradiction becomes a named event instead of a difference that dissolves inside a fused representation. Resolution runs against a table declared in advance and indexed by claim type, not by channel alone: what was literally said is settled by the transcript, how it was said by the acoustics, what was physically present by the frames, what an interface element is called by the accessibility tree. Where the table has an entry, the authoritative channel is applied and the losing claim is kept next to the decision as the reason the answer is uncertain. Where it has no entry, or where the authoritative channel is itself low-confidence, the agent abstains or escalates instead of picking. A reflection step revisits only the disagreeing slots, which keeps the extra reasoning proportional to the conflict rather than to the whole observation.

When to use

• One situation is observed through two or more channels that can each support a claim independently, such as transcript and acoustics, screenshot and accessibility tree, or caption and frames.
• Disagreement between the channels is itself informative — sarcasm, dubbing, editing, mislabelled interface elements, stale document text.
• A wrong answer costs more than a late one, so abstaining or escalating on an unresolvable conflict is acceptable.
• Domain experts can state in advance which channel is authoritative for which kind of claim.