AlphaWiSE: Adaptive Weight Interpolation for Continual Multimodal Representation Learning
Computer Vision and Pattern RecognitionMachine Learning
Summary
The authors study how multimodal models like CLIP, which connect different types of data such as images, text, and audio, can lose their ability to relate these modes well when updated continuously with new data. They propose AlphaWiSE, a method that combines two existing model versions by carefully blending their parameters using a simple formula learned from a small set of examples. This approach produces a single model that keeps the benefits of both versions without extra cost during use. Their experiments show that AlphaWiSE improves various types of cross-modal retrieval better than other continual learning methods.
Authors
Sarthak Jain, Qiran Hu, Zhen Zhu, Yaoyao Liu
Abstract
Multimodal models such as CLIP learn a shared embedding space for cross-modal retrieval, but continual adaptation to sequentially arriving data can disrupt the cross-modal alignment acquired from earlier phases. Conventional continual-learning methods return a single checkpoint, which commits every retrieval direction to the same stability-plasticity trade-off. We propose AlphaWiSE, a post-hoc weight-space interpolation method that composes two frozen source checkpoints. For each aligned parameter tensor identified by its checkpoint key, AlphaWiSE fits one scalar interpolation coefficient shared by all tensor entries. The coefficients are fitted on a smaller exemplar memory and used to materialize one interpolated checkpoint. The deployed model has the same architecture and parameter count as either source checkpoint, which does not require additional inference time. Extensive experiments on audio-image-text retrieval show consistent improvements over strong continual-learning baselines across multiple retrieval directions and evaluation metrics.