Scaling laws reveal modality impacts on multimodal AI training
Scaling Laws for Multimodal Data Mixtures
Computation and LanguageMachine Learning
Summary
Training AI models that understand text, images, and sounds together is tricky because these different types of data can interfere with each other. The authors studied how much of each type of data to use and how big the models should be for the best results. They found that audio needs larger models more than more data, text and vision need a balance, and repeating image or audio data too much doesn't help. Also, images interfere with audio learning a lot, but audio doesn't interfere much with text or images. These findings help AI builders mix different types of data more effectively.
What this means in practice
- •For ai model engineers: Adjust training data proportions and model sizes based on specific modalities to optimize multimodal model performance and reduce interference.
- •For cloud ai platform teams: Design multimodal training workflows that avoid excessive repetition of vision or audio data to save compute without losing accuracy.
Authors
Aditi Khandelwal, Ayush K. Tarun, Yixuan Xu, Imanol Schlag, Golnoosh Farnadi, Siva Reddy, Luke Zettlemoyer, Antoine Bosselut
Abstract
Frontier AI systems are increasingly natively multimodal, jointly pretrained on multiple modalities, such as text, vision, and audio. However, the scientific understanding of the multimodal data mixtures used to pretrain these systems --- how much of each modality to mix, and at what potential interference cost to the others --- is largely absent or gatekept. In this work, we derive data mixing scaling laws for three modalities: text, vision, and audio. Multimodal Large Language Models (MLLMs) increasingly adopt Mixture-of-Experts (MoE) architectures, as MoEs scale efficiently and partially limit cross-modal interference. We therefore use MoEs to conduct 268 experiments ranging from 457M to 8.3B parameters, on up to 150 billion tokens. Our work leads to multiple actionable insights into multimodal data mixing for training MLLMs. First, compute optima are modality-specific: audio modality is model-heavy, favoring scaling parameters over tokens, while text and vision require balanced scaling of model size and data. Second, repeating vision or audio data beyond 2x yields negligible benefits. Finally, cross-modal interactions are highly asymmetric: vision strongly interferes with audio, whereas audio's interference with learning text and vision is negligible. Overall, we provide a principled foundation for understanding the multimodal data mixtures needed to train frontier MLLMs.