Synco improves learning from combined data using interaction signals
SynCo: Learning Cross-Modal Synergy by Contrasting Interaction Residuals
Computer Vision and Pattern RecognitionMachine Learning
Summary
Different types of data, like images and text, each have unique and shared information. The authors point out that existing methods mostly learn from information that is shared across data types and often miss the important combined signals that emerge only when considering the data together. They introduce SynCo, a new method that highlights this unique combined information by focusing on what cannot be predicted from each type alone. This approach improves performance on both synthetic tests and real-world tasks.
What this means in practice
- •For machine learning engineers: Enhance multimodal models to better capture information unique to combined data inputs, improving performance in tasks like multimedia classification or sensor fusion.
- •For data integration teams: Improve methods for integrating diverse data sources by focusing on interaction signals that are missed by standard techniques.
Authors
Yavuz Yarici, Ghassan AlRegib
Abstract
Multimodal contrastive learning is a dominant paradigm for learning transferable representations from unlabeled data, but standard objectives primarily capture information that is redundant between modalities. Partial Information Decomposition (PID) shows that task-relevant information in multimodal data decomposes into three components: redundancy shared between modalities, uniqueness specific to each modality, and synergy available only from their joint observation. Recent frameworks extend contrastive learning to capture all three components, yet synergy remains undertrained in practice. We propose SynCo (Synergy Contrastive Learning), a method that directly addresses synergy undertraining through dedicated supervision on an interaction residual. SynCo fits a linear projector to predict the fused representation from independently computed unimodal features, and the resulting interaction residual, which removes the linearly unimodal-predictable component, receives dedicated contrastive supervision at negligible computational cost. On the controlled Trifeature benchmark, SynCo achieves state-of-the-art synergy capture with a $+5.98\%$ gain over the baseline, and on real-world benchmarks from MultiBench, DARai, and MM-IMDb, SynCo consistently outperforms or matches prior methods across diverse modality combinations and task types. The method operates as a plug-in to existing contrastive multimodal frameworks without modifying the underlying fusion architecture and can further improve synergy capture when combined with other methods.