Vision capacity matters more than pretrained encoders in multimodal models
Unbundling Multimodal Vision Encoding
Computer Vision and Pattern RecognitionMachine Learning
Summary
When teaching computers to understand pictures and language together, it’s unclear how to best process the images. The authors showed that a special visual processing part, even if trained from scratch, works as well as one pretrained separately. What really helps is having dedicated computing power just for vision tasks inside the model. Also, where this vision part is placed matters less, as it can be added early or later and still perform well.
What this means in practice
- •For ai model builders: Design multimodal models with joint training of vision capacity to improve efficiency and maintain visual understanding without separate pretraining stages.
- •For machine learning engineers: Incorporate vision modules early within the backbone network to reduce inference time while preserving visual question answering accuracy.
Authors
Ellis Brown, Chris Hoang, Shengyi Qian, Shang-Wen Li, Mike Lewis, Marjan Ghazvininejad, Luke Zettlemoyer, Lucas Beyer, Mengye Ren, Jakob Verbeek, Saining Xie, Hu Xu
Abstract
How multimodal language models should encode visual input remains up for debate. Most models use a pretrained vision encoder, while some recent recipes train the encoder from scratch with the language model or remove it entirely. Vision encoders bundle together three design choices: a visual prior from upstream pretraining, dedicated vision capacity, and pre-fusion placement of that capacity (i.e., before the multimodal backbone). Because published models also differ in data, architecture, and compute, comparing them cannot isolate the effect of each choice. We disentangle these choices via a controlled framework where we pretrain all models from scratch on common data and compare them on visual question answering and language benchmarks at matched lifetime compute. We find that the visual prior is not necessary; an encoder trained jointly from scratch is on par with a pretrained one, without a separate training stage. What matters instead is dedicated vision capacity: an encoder-free model trails on visual understanding even when its backbone is enlarged by the encoder's parameter count. Yet that capacity need not precede multimodal fusion, as vision-only modules added in-fusion to early backbone layers perform comparably to a pre-fusion encoder and can lower time to first token. These results also hold across backbone sizes, when encoder pretraining is treated as free, and when starting from a pretrained language model. Overall, our findings favor dedicated vision capacity trained jointly with the backbone and placed early within it.