Dynamic multimodal learning combines multiple experts to avoid forgetting
NeuCME: Toward Dynamic Multimodal Continual Learning via Neural Combinatorics of Multiple Experts
Machine LearningComputer Vision and Pattern Recognition
Summary
When teaching machines to learn from different types of information like images and sounds, it is hard for them to keep learning new things without forgetting the old ones, especially when the types of information change each time. The authors created a new method called NeuCME that helps machines remember past knowledge and combine information from different sources effectively, even when these sources vary between tasks. They tested this method on real-world datasets and found it works better than current approaches. This advances the ability to build smarter systems that learn continually from changing data.
What this means in practice
- •For software engineers: Improve AI models in applications requiring ongoing learning from changing combinations of data sources without retraining from scratch.
- •For robotics developers: Develop robots that adapt continuously as they interact with environments involving different sensor types over time.
Authors
Kai Guo, Chuanbin Liu, Peng Hu, Hao Wang, Xi Peng
Abstract
Multimodal continual learning has recently shown great potential for developing agents with human-like intelligence by continuously learning new tasks across multiple modalities. However, existing methods typically assume that the set of modalities per task is predefined and fixed. In this paper, we investigate a more realistic learning setting, referred to as dynamic multimodal continual learning, in which the set of modalities may vary across tasks rather than remaining fixed. This setting involves two primary challenges: (i) spatio-temporal catastrophic forgetting and (ii) adaptive multimodal fusion. To address these challenges, we propose NeuCME (as shorthand for \textbf{Neu}ral \textbf{C}ombinatorics of \textbf{M}ultiple \textbf{E}xperts), a novel framework designed to effectively learn and integrate knowledge across tasks with varying modalities. The proposed NeuCME model comprises three key components, namely modality-combinational rehearsal, multi-gated mixture-of-experts, and task relevance-guided distillation. Furthermore, we formulate an evaluation metric to quantify the dynamism of task sequences and then set up a comprehensive benchmark with different degrees of dynamism. Extensive experiments using four real-world datasets demonstrate that the proposed NeuCME outperforms state-of-the-art methods markedly.