OmniRoute improves efficiency in multimodal audio visual models

OmniRoute: Mapping Temporal Semantic Evidence to Audio-Visual Token Budgets for Efficient Omnimodal Large Language Models

Computer Vision and Pattern Recognition

Summary

Processing both audio and video together in language models takes a lot of computing power, especially when handling long clips. The authors designed OmniRoute to reduce this workload by smartly choosing what parts of the audio and video to keep based on how important they are over time. It works in two steps: first, it figures out which modality (audio or video) matters more at different times, then it compresses that data accordingly while adjusting the other. This results in faster processing without losing much understanding.

What this means in practice

  • For multimedia app developers: Reduce processing time and resource use when building apps that interpret both audio and video inputs in real time.
  • For streaming service engineers: Improve efficiency of analyzing live video and audio streams for content moderation or metadata extraction by prioritizing important segments.

Authors

Yuchen Deng, Zidang Cai, Feidiao Yang, Yufei Wang, Jie Wang, Hai-Tao Zheng, Yuxing Han

Abstract

Omnimodal large language models (Omni-LLMs) encode audio and visual streams into temporally interleaved token sequences for multimodal reasoning. However, processing long audio-visual token sequences incurs substantial prefill costs. Existing compression methods have made progress, but often overlook temporal changes in audio-visual semantic relevance. Motivated by temporal variation and local continuity, we propose OmniRoute, a training-free, two-stage compression framework. First, Temporal Evidence-Guided Budgeting (TEGB) derives chunk-wise modality preferences and initial leading-modality budgets from semantic relevance and local content variation. Second, Budget-Constrained Semantic Compression (BCSC) compresses the leading modality and then calibrates the follower's retention target using the actual retained fraction. For video, it combines spatiotemporal grouping with query-guided selection; for audio, it selects tokens based on encoder attention and query relevance, then merges residual tokens into context anchors under visual guidance. Experiments on four representative benchmarks demonstrate a better trade-off between inference efficiency and performance than competitive baselines. The code and interface will be released to facilitate further research.