Text training boosts reasoning in multi-modal large language models
Text-Centric Post-Training for Omni-Modal Reasoning
Computation and Language
Summary
Multi-modal large language models that understand both audio and visuals can struggle with complex reasoning that involves multiple steps. The authors found that training these models using just text helps improve their reasoning abilities significantly, even without audio-visual data. However, focusing only on text harms the model's ability to perceive audio and visuals accurately. To fix this, they propose a two-step training approach: first, train mainly on text for reasoning, then fine-tune with a small amount of audio-visual data to restore perception. This method greatly improves reasoning while keeping perception strong and reduces training costs.
What this means in practice
- •For ai model developers: Enhance multi-modal AI reasoning performance by focusing initial training on text before refining audio-visual perception, reducing training time and computational cost.
- •For speech recognition engineers: Improve audio-visual systems by applying text-centric post-training to boost multi-step reasoning while maintaining accurate perception with less data and compute.
Authors
Ziyang Cheng, Yuhao Wang, Hongcheng Liu, Qimin Wu, Jingru Fan, Chen Qian, Yanfeng Wang, Yu Wang
Abstract
Improving joint audio-visual reasoning in Omni Large Language Models typically incurs substantial data construction and training costs. Our diagnostics reveal multi-hop reasoning difficulties despite correct answers to all corresponding single-hop questions and suggest partial decoupling in the local optimization of perception and reasoning objectives. This motivates post-training with different emphases on these capabilities. Text-only reasoning training yields gains across data sources, model scales, and families. With the best-performing text-only configuration, supervised fine-tuning followed by reinforcement learning (RL) raises Qwen2.5-Omni-7B's geometric mean of nine reasoning scores by 25.83% over the base model, outperforming the complete native audio-visual route with 56.6% fewer GPU-hours. Training on data synthesized entirely by a text-only LLM raises this geometric mean by 21.01% without audio-visual data in construction or training. However, text-only training degrades perception. We therefore propose a text-centric post-training paradigm: text-only training provides the main reasoning optimization, and reduced-data native audio-visual RL then refines perception. Refinement uses about 90% fewer input tokens than full-data audio-visual RL, restores perception above the base level, and retains 93.5% of the best-performing text-only pipeline's reasoning gain.