Multimodal alignment strategies win differently by task and data budget
Joint, Decoupled, or Agent: When Each Multimodal Alignment Paradigm Wins
Artificial IntelligenceComputation and LanguageComputer Vision and Pattern RecognitionMultimedia
Summary
Combining information from different sources like images and text can be done in three ways: training everything together, training separately then combining, or using a big language model with instructions. This research compared these methods across many tasks and data sizes to find when each works best. No single way always wins; it depends on the task and how much labeled data is available. They also developed a way to predict which method to use based on task features without trying them all.
What this means in practice
- •For machine learning engineers: Choose the best multimodal fusion method based on task type and data size before model training to improve performance.
- •For ai product managers: Plan data labeling efforts and model deployment by predicting which multimodal alignment strategy maximizes utility for given tasks.
Authors
Yi Fang, Gaurav Srivastava, Sirui Ding, Xuan Wang
Abstract
Jointly pretrained multimodal encoders are a common way to combine modalities, while alternatives integrate their information during downstream training or inference. When does each approach win? We compare three paradigms, instantiated with public checkpoints, across 16 paired datasets in five domains and multiple training-label budgets. JOINT trains a fusion head on frozen, jointly pretrained encoders. DECOUPLED trains a fusion head on frozen, independently pretrained encoders. AGENT gives a frozen LLM task instructions, labelled examples, available sample text, and supervised unimodal readouts. No paradigm dominates the complete 144 dataset-budget combinations: DECOUPLED wins 54, JOINT 44, and AGENT 46 by mean utility. ScienceQA, MELD, and MetMeme shift from AGENT to fusion as supervision increases. Between the fusion paradigms, JOINT leads on four of five QA datasets across all evaluated budgets, while higher unimodal 5-NN accuracy is associated with a relative shift toward DECOUPLED across eight classification datasets (Spearman $ρ=-0.67$). We test whether representation and task diagnostics predict these preferences on held-out datasets before fitting their candidate systems. A two-step cascade combines these diagnostics with task and budget metadata to select AGENT or fusion, then JOINT or DECOUPLED. Under leave-one-group-out evaluation, it selects the winner in 71.4% of cells versus 54.2% for metadata-only routing, with mean relative regret 0.028 versus 0.057. Using geometry, supervised representation diagnostics, and text-label matching together yields higher accuracy and lower regret than using any one of these sources with metadata. These results connect paradigm preferences to supervision budget, task format, and unimodal 5-NN accuracy. Target-task diagnostics thus support selecting among the three paradigms before candidate fitting.