Tracing influence to merge experts in multimodal AI models
Tracing and Coordinating Cross-Layer Influence for Multimodal Model Merging
Artificial Intelligence
Summary
Combining different AI experts into one model is tricky because changes in one part affect later parts, especially when mixing vision and language skills. The authors propose a method called TAC-Merge to track how each expert's changes spread across layers and work together better. Their approach uses special math tools to map these effects and finds the best way to merge experts while keeping their unique strengths. Tests show this method improves merged models' performance on various tasks, even new ones.
What this means in practice
- •For multimodal ai developers: Create models that combine vision and language experts more effectively for broader task performance by tracing and coordinating their influence across layers.
- •For machine learning engineers: Merge specialized AI models into one unified model without losing complementary capabilities by jointly optimizing how expert updates interact.
Authors
Pengyang Zhou, Xiaobin Tu, Zhengxi Liu, Rongkun Xue, Haochen Li, Miancan Liu, Ziyuan Chen, Yinggui Wang, Jinkui Ren, Xiantao Zhang
Abstract
Multimodal model merging aims to consolidate task experts into a single model that retains their complementary capabilities. Most unimodal model merging methods combine expert updates within individual layers, and multimodal approaches largely follow this design. However, an expert update changes the representations passed to subsequent layers, allowing its influence to propagate across depth and affect how visual and textual information interact. When visual and language updates are combined, later updates act on inputs already modified by earlier ones, coupling their effects. This poses two challenges: (1) how to characterize the multimodal influence of individual expert updates across depth, and (2) how to jointly combine expert updates based on their multimodal influence. To address these challenges, we propose TAC-Merge for tracing and coordinating cross-layer influence in multimodal model merging. It contains two modules, i.e., multimodal influence mapping (MIM) and coupled merge control (CMC). MIM constructs graphs of update effects and uses Ricci curvature together with expert predictions to define a shared fusion objective. CMC models interactions among coefficient adjustments and jointly optimizes regional weights to synthesize one shared model. Experiments across diverse multimodal tasks demonstrate the effectiveness of TAC-Merge in consolidating complementary expert capabilities and supporting generalization to unseen tasks.