Omni-modal large language models improve sharing knowledge from multiple senses
Omni-Modal LLMs Know More Than They Say
Computation and LanguageArtificial IntelligenceComputer Vision and Pattern Recognition
Summary
Large language models that can understand more than just text, like images and sounds, often know things from these extra inputs but don’t talk about them well. The authors found a way to measure how well different senses connect to the words the model generates. They then created a method to help these models explain what they see or hear more clearly by turning their perceptions into text first. This method made the models better at sharing knowledge from all their inputs, not just text.
What this means in practice
- •For multimodal ai developers: Improve the clarity and completeness of outputs when building AI systems that handle images, audio, or sensor data alongside language.
- •For human-computer interaction teams: Create more balanced AI assistants that effectively share knowledge from multiple sensory inputs for richer user interactions.
Authors
Na Min An, Taekyung Kim, Jaemyung Yu, Hyunjung Shim, Dongyoon Han
Abstract
Omni-modal large language models (Omni-LLMs) extend LLMs to process heterogeneous modalities (e.g., vision, audio, and IMU signals) alongside language, enabling a broader range of multimodal tasks. Yet we find that these models often fail to express information from non-textual modalities even when they perceive it correctly. We term this disconnect knows-but-cannot-say. To investigate this phenomenon, we introduce Token-level Modality Alignment (TMA), which measures the alignment between individual modality representations and response representations. Our analysis shows that non-textual representations are less aligned with response tokens and receive less attention than textual representations, suggesting a potential explanation for uneven modality preferences. Motivated by these findings, we propose OmniCatalyst, a simple perceive $\rightarrow$ infer strategy that complements the original modality inputs with self-generated text, making non-textual knowledge more accessible. Across seven Omni-LLMs, OmniCatalyst consistently improves performance (+7.29%) and yields more balanced modality preferences, with further gains from diverse training strategies. Further analysis demonstrates that the knowledge extraction phases increase TMA, suggesting that improved alignment makes non-textual information more accessible during response generation. In summary, our findings uncover how Omni-LLMs can better express modality knowledge through language. Our code is available at https://github.com/naver-ai/OmniLLM.