Omni modal large language models improve with self teaching on compressed data
Learning to Reason with Compressed Context: Ground-Truth-Free Adaptation of OmniLLMs via Self-Distillation
Computer Vision and Pattern Recognition
Summary
Long sequences of audio and video information make large language models slow and costly to use. The authors introduce a way to help these models learn to understand compressed data better, without needing extra labeled examples or correct answers. Their method uses the full, uncompressed data view to teach the model how to work well with the shorter, compressed version. Tests show this approach improves accuracy across many tasks, helping balance speed and performance.
What this means in practice
- •For mobile app developers: Deploy audio-video understanding features on devices with limited computing power by adapting large models to compressed inputs.
- •For streaming platform engineers: Improve video and audio content analysis accuracy while using less computational resources by applying self-distilled compressed context adaptation.
Authors
Jianghao Wang, Ke Meng, Jian Li, Chi Cheng, Longyu Qi, Liyin Liang, Yifeng Qian, Chunbo Lai, Yutian Lin, Zeyu Wang
Abstract
Omni-modal large language models (OmniLLMs) enable unified audio-video understanding, but their long multimodal token sequences make deployment computationally expensive. Token compression reduces this cost, yet aggressive compression often lowers accuracy. Existing works predominantly focus on designing better compression mechanisms; however, adapting the underlying language model to reason effectively over the remaining compressed context remains under-explored. To address this, we propose CAFD (Compressed-Context Adaptation via Full-Context Distillation), a ground-truth-free self-distillation framework that adapts OmniLLMs to fixed compression pipelines without requiring reference answers, rationales, or correctness rewards. CAFD leverages the full-token view of the same multimodal sample as a source of privileged information: a full-context self-teacher provides soft target supervision to a compressed-context student along the student's on-policy trajectory. Evaluated on Qwen2.5-Omni-7B across five audio-video benchmarks, five compression pipelines, and five deployment budgets, CAFD demonstrates consistent gains, improving 120 out of 125 conditions with an average accuracy boost of 1.44 points and recovering 26.9% of the accuracy gap on average. These results demonstrate that the proposed ground-truth-free adaptation offers an effective and practical route to improving the accuracy-efficiency trade-off in deployed OmniLLMs.