Adaptive Hierarchical Representation Alliance for Multimodal Learning
2026-08-24 • Multimedia
Multimedia
AI summaryⓘ
The authors found that combining language, vision, and audio in one final layer assumes these different types of information become meaningful at the same depth, which is not true. Text usually needs deeper understanding, while images and sounds give useful clues earlier. To fix this, they created AHRA, a method that separates shared and unique information at different levels and uses special modules to better combine and select important features. Their approach improved performance consistently across multiple tests, especially when some inputs were noisy or missing.
multimodal modelssemantic granularitylayer-wise CKAshared-private representationcross-modal expertsoft gatinghierarchical fusionmultimodal intent recognitionsentiment analysisnoise robustness
Authors
Chunlei Meng, Pengbin Feng, Jacqueline J. Pang, Chih-Ting Liao, Rong Fu, Zhaolu Kang, Zhongxue Gan, Chun Ouyang
Abstract
Multimodal models often align language, vision, and audio in a single final-layer latent space, implicitly assuming that task-relevant evidence emerges at the same semantic depth across modalities. Using layer-wise CKA analysis, we observe that this assumption leads to semantic granularity mismatch: textual cues usually require deeper contextual abstraction, whereas visual and acoustic cues often provide discriminative perceptual evidence in shallow or middle layers. This mismatch can flatten fine-grained modality-private cues and reduce reliability under noisy, imbalanced, or missing inputs. To address this, we proposed Adaptive Hierarchical Representation Alliance (AHRA), a hierarchical shared--private expert framework. AHRA factorizes each modality into shared and private streams across semantic levels, regularizes them with shared alignment and private decorrelation, routes shared information through a cross-modal expert, and enhances task-relevant private tokens with modality-specific experts guided by a sparsity-controlled soft-gating mechanism (foreground exam). A hierarchical co-fusion module then performs intra-level expert coordination and inter-level semantic selection. Experiments on six benchmarks across image-text classification, multimodal intent recognition, and trimodal sentiment analysis show that AHRA consistently improves over strong baselines and remains robust under noisy and missing-modality settings.