AI summaryⓘ
The authors show that instead of combining features from different types of data (modalities) into one model, using several separate models for each type and then combining their predictions works better, especially when the data types are very unbalanced. They compare these 'deep ensembles' to other fusion methods and find ensembles consistently perform better with the same number of parameters. They also suggest a simple way to decide how many models to use for each data type in the ensemble, which aligns with their experiments. To test their ideas systematically, they built a synthetic setup controlling the difficulty and data types, confirming their results on both made-up and real data. Lastly, they use scaling laws to predict how well these ensembles can do as they get bigger.
multimodal classificationlate fusionunimodal neural networksdeep ensemblesmodality imbalanceintermediate fusionscaling lawssynthetic multimodal frameworkpredictive strengthhybrid methods
Authors
Ilya Burenko, Dmitry Vetrov
Abstract
In multimodal classification, late-fusion approaches classify concatenated modality-specific features extracted by unimodal neural networks. When modality imbalance is pronounced, various regularization techniques have been proposed to balance the learning process and overcome the inferior performance of late-fusion networks. In contrast, this work demonstrates that multimodal data can be effectively classified without any explicit modality fusion, using deep ensembles of unimodal networks. We systematically compare deep ensembles to late-fusion networks at equal parameter count and show that ensembles consistently outperform state-of-the-art late-fusion methods designed to address modality imbalance. This advantage also holds over intermediate-fusion techniques we evaluated and over hybrid methods that combine unimodal and multimodal predictions. We propose and empirically validate a method for selecting the number of models per modality in an ensemble, avoiding computationally expensive exhaustive search. Under extreme modality imbalance and small ensemble sizes, the heuristic indicates that ensembles of unimodal models trained solely on the stronger modality are preferable; as the ensemble scales up, incorporating models from the weaker modality becomes beneficial. Both predictions align with our empirical findings. To systematically explore the challenges of optimizing multimodal models, we propose a synthetic multimodal framework that allows control over both the number of modalities and their predictive strength; our findings are consistent across synthetic and real-world datasets. Finally, by fitting scaling laws to bimodal datasets, we estimate the asymptotic performance of ensembles.