Hyper-llava improves multimodal task routing with uncertainty awareness
Hyper-LLaVA: Hyperbolic Uncertainty-aware Modality-Balanced Routing for Multimodal Continual Instruction Tuning
Computer Vision and Pattern Recognition
Summary
Multimodal Continual Instruction Tuning helps AI systems learn tasks that involve different types of data, like images and text, over time. The authors found that current methods don't use all the information inside each data type well, and some types of data can confuse the system. They created Hyper-LLaVA to measure uncertainties and balance the importance of each data type more effectively. This new approach enables better matching of tasks and improves the system’s routing decisions, resulting in stronger overall performance.
What this means in practice
- •For multimodal ai developers: Enhance task routing accuracy in systems that process multiple data types over time by integrating uncertainty-aware cross-modal balancing.
- •For ai system integrators: Improve continual instruction tuning of multimodal AI to better handle diverse and evolving task demands without degrading performance.
Authors
Kunlun Xu, Yanqin Zhang, Wenwen Qiang, Jiahuan Zhou
Abstract
Multimodal Continual Instruction Tuning (MCIT) aims to exploit the incrementally accumulated knowledge to process multimodal inputs of diverse tasks, where parameter routing plays an important role. State-of-the-art methods rely on sample-to-task center similarity and cross-modal fusion with equal weight during routing. However, such solutions face two fundamental flaws: (1) Within each modality, the sample-to-task center distance is sub-optimal for routing since the abundant intra-task diversity information is underleveraged. (2) Different modalities exhibit varying reliability across tasks, where the modality with inter-task ambiguity can easily misguide the routing result. To address these problems, we propose Hyperbolic Uncertainty-aware Modality-Balanced Routing (Hyper-LLaVA) to improve parameter routing capacity based on cross-modality task feature uncertainty modeling. Specifically, to improve intra-modality task matching, Hyper-LLaVA accesses the sample-to-task distribution similarity in the Hyperbolic space. Besides, to alleviate the degradation brought by unreliable modalities, Hyper-LLaVA quantifies the task matching ambiguity within each modality to achieve adaptive balancing between task matching across modalities. Based on the complementary intra- and inter-modality task matching enhancement, our Hyper-LLaVA outperforms state-of-the-art approaches by large margins. Our source code is available at https://github.com/zhoujiahuan1991/ICML2026-Hyper-LLaVA