Multimodal language models controlled to trust specific inputs
Beyond Modality Preference: Controlling Multimodal Arbitration under Modality Conflict in MLLMs
Computer Vision and Pattern Recognition
Summary
Multimodal large language models (MLLMs) combine information from different types of input, like images and text, but can get confused when these inputs disagree. The authors found certain parts of the model, called arbitration heads, that decide which kind of input the model listens to more. They created a method that lets users control this decision at inference time, without needing extra training. This helps the model focus on the most reliable input for a task, improving its predictions.
What this means in practice
- •For multimodal ai developers: Control model attention to trusted input types when training data has conflicting modalities.
- •For product designers using ai assistants: Customize AI responses to prioritize either image or text inputs depending on specific user needs or contexts.
Authors
Zhaoying Pan, Shenyu Lu, Jun Hui Wong, Junyi Chai, Xiaoqian Wang
Abstract
Multimodal large language models (MLLMs) integrate information across modalities, but become unreliable when those modalities disagree. Under conflicting inputs, MLLMs may favor modalities that are not the most reliable for the task, thus motivating inference-time control of modality arbitration. We identify a small set of attention heads, termed arbitration heads, that causally influence whether predictions follow visual or textual evidence. Building on this, we introduce a lightweight, training-free intervention that biases these heads toward a designated trusted modality at inference time. The identified heads are stable across calibration set sizes and generalize across settings, and steering them enables effective bidirectional control of modality reliance across diverse benchmarks. Our findings identify an effective control mechanism to align MLLM predictions with task-specified modality priorities. The code is available at https://github.com/zhaoyingpan/modality-arbitration.