MultiLoReFT: Decoupling Shared and Modality-Specific Subspaces in Multimodal Learning via Low-Rank Representation Fine-Tuning
Machine Learning
Summary
The authors explain that combining different types of information, like images and text, is important but hard to do because good paired data is rare and existing methods mix shared and unique information in a confusing way. They created MultiLoReFT, a new technique that uses small updates on already trained single-modality models to learn clear patterns separating shared info from modality-specific details. Their approach works well on both fake and real examples, making the combined information easier to understand and use for decisions.
Authors
Sana Tonekaboni, Viktoria Schuster, Caroline Uhler
Abstract
Real-world perception and decision making are inherently multimodal, integrating complementary signals across modalities. However, training multimodal models faces two main obstacles. First, collecting large-scale, well-aligned paired multimodal datasets is often impractical, making end-to-end multimodal training difficult. Second, existing multimodal representations frequently entangle information shared across modalities with modality-specific information, hindering interpretability and control. We introduce MultiLoReFT, an efficient and scalable low-rank representation fine-tuning framework for multimodal learning with pretrained unimodal models. MultiLoReFT extends low-rank adaptation to the multimodal setting and learns interpretable projection subspaces that decouple shared and modality-specific information. Across simulated and real-world benchmarks, it produces representations that support multimodal prediction while explicitly revealing how shared and modality-specific information is distributed across modalities.