Omni-modal language models wrongly rely on images for audio answers

Seeing What Should Be Heard: Diagnosing and Repairing Cross-Modal Shortcuts in Omni-Modal LLMs

Computer Vision and Pattern RecognitionComputation and LanguageMachine Learning

Summary

Omni-modal language models are designed to answer questions using the specific type of input mentioned, like audio or images. The authors found that when asked audio questions, these models often use the related image instead of the audio, which can lead to errors. To figure out how much the models rely on each input, they created a way to swap images and audio between samples and see which one matters more. They then developed a training fix that teaches models to focus on the right input, which reduced the models' overreliance on images without hurting their ability to answer audio questions.

What this means in practice

  • For multimodal ai engineers: Improve training pipelines to prevent models from relying on irrelevant image data when answering audio questions.
  • For voice assistant developers: Enhance voice assistants to better focus on audio signals rather than correlated visuals when interpreting user queries.

Authors

Yueran Ma, Ronghao Lin

Abstract

Omni-modal large language models (LLMs) are expected to answer a question using the modality it explicitly refers to. However, existing training paradigms rarely verify whether models actually follow this modality, because multimodal inputs from the same sample often provide redundant evidence for the same answer. In this work, we uncover a pervasive cross-modal shortcut in omni-modal LLMs: when asked an audio-related question, models rely on the image as much as on the audio, and sometimes even more. To systematically diagnose this behavior, we introduce the Factorized Modality Diagnostic, which independently swaps audio and images between samples to isolate each modality's causal contribution. Across two model families in different settings, we find that this shortcut persists throughout supervised fine-tuning and reinforcement learning post-training, while judge-based RL may further amplify such reliance on irrelevant visual information. Based on this finding, we propose DMC-Repair, which trains models on the same kind of cross-modal swapped samples while assigning supervision according to the modality specified by the question. This prevents models from exploiting the spurious correspondence between modalities within the same clip. Experiments demonstrate that DMC-Repair reduces the image-induced share of the answer effect by 59.9%, effectively suppressing the cross-modal shortcut without compromising audio-question answering performance. The reduction in shortcut reliance generalizes across two model families and zero-shot to an unseen dataset and an unseen benchmark, and persists through subsequent post-training. Code is available at https://anonymous.4open.science/r/DMC-Repair.