Multimodal learning improves weaker signals without hurting strong ones
Probe-Guided Gradient Balancing for Multimodal Learning
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
When computers learn from data that comes in several forms, like images and words together, the parts that learn faster often overshadow the weaker parts. The authors present a technique called Probe-Guided Gradient Boosting that detects which parts are weaker and helps them learn better without slowing down the stronger parts. They do this by attaching simple tools that check each data type separately and then balance the learning signals. Tests show this method works better on tricky cases where some data types are much harder to learn from.
What this means in practice
- •For machine learning engineers: Improve training of models that combine different types of data by better balancing learning speed across data sources.
- •For audio-visual system builders: Enhance multimodal systems that fuse sound and images by preventing dominant modalities from suppressing weaker ones.
Authors
Vu Vo, Haytham M. Fayek, Thuy Nguyen
Abstract
Jointly trained multimodal networks frequently underutilize weaker modalities, as faster-learning modalities dominate the shared fusion objective and suppress gradients reaching slower ones. Most existing approaches throttle dominant modalities, whereas directly boosting weaker modalities often fails because monitoring signals derived from the joint loss remain dominated by stronger modalities. We propose Probe-Guided Gradient Boosting (PGGB), a gradient modulation approach that boosts weaker modalities while preserving the strength of dominant ones. PGGB attaches lightweight linear classifiers to stop-gradient features from each modality to estimate representation quality independently. The gap between these scores, termed the utilization gap, provides an imbalance signal used to adaptively scale weaker-modality gradients through bounded and smoothed modulation. PGGB is composable with throttling methods, as it leaves the loss function and fusion architecture unchanged. Experiments across eight benchmarks spanning four domains and 2-4 modalities demonstrate that PGGB outperforms state-of-the-art methods on highly imbalanced multimodal datasets while remaining competitive under low or negligible imbalance. We further establish bounded scaling, self-attenuation, and a descent bound for standard SGD under smoothness and bounded-variance assumptions. Code is available at https://github.com/VuVoCodes/PGGB.