NeuCME improves learning across changing combinations of data types
NeuCME: Toward Dynamic Multimodal Continual Learning via Neural Combinatorics of Multiple Experts
Machine LearningComputer Vision and Pattern Recognition
Summary
Learning from different types of information, like images and sounds, can be hard when the types change from task to task. Existing methods usually expect the same kinds of data every time, but real-world learning is more flexible and unpredictable. The authors study this realistic scenario and identify two main problems: forgetting past information in space-time and combining new data types effectively. They propose NeuCME, a method with three components that helps a system remember and fuse knowledge from various combinations of data. Testing on real datasets shows their approach works better than previous methods.
multimodal learningcontinual learningcatastrophic forgettingmodality fusionmixture-of-expertsknowledge distillationmachine learningspatio-temporal data
Authors
Kai Guo, Chuanbin Liu, Peng Hu, Hao Wang, Xi Peng
Abstract
Multimodal continual learning has recently shown great potential for developing agents with human-like intelligence by continuously learning new tasks across multiple modalities. However, existing methods typically assume that the set of modalities per task is predefined and fixed. In this paper, we investigate a more realistic learning setting, referred to as dynamic multimodal continual learning, in which the set of modalities may vary across tasks rather than remaining fixed. This setting involves two primary challenges: (i) spatio-temporal catastrophic forgetting and (ii) adaptive multimodal fusion. To address these challenges, we propose NeuCME (as shorthand for \textbf{Neu}ral \textbf{C}ombinatorics of \textbf{M}ultiple \textbf{E}xperts), a novel framework designed to effectively learn and integrate knowledge across tasks with varying modalities. The proposed NeuCME model comprises three key components, namely modality-combinational rehearsal, multi-gated mixture-of-experts, and task relevance-guided distillation. Furthermore, we formulate an evaluation metric to quantify the dynamism of task sequences and then set up a comprehensive benchmark with different degrees of dynamism. Extensive experiments using four real-world datasets demonstrate that the proposed NeuCME outperforms state-of-the-art methods markedly.