LoopVL improves vision language tasks with recurrent updates
LoopVL: Recurrent Visual Intelligence
Computer Vision and Pattern RecognitionComputation and Language
Summary
Understanding images and language together is tricky because they are very different types of information. The authors created LoopVL, a model that repeatedly updates its understanding by looping through the same vision-language processes multiple times. This improves the model's ability to reason about images and language compared to other models of similar or larger sizes. They also observed interesting patterns in how the model's visual attention changed during these loops, showing it refines its focus over time.
What this means in practice
- •For mobile app developers: Build apps that understand images and text better by using a recurrent model for improved visual reasoning and multimodal understanding.
- •For robotics teams: Enhance robot perception systems by integrating recurrent vision-language models that iteratively refine scene interpretation for complex tasks.
Authors
Zhe Qian, Ziyang Gong, Zhongxing Xu, Hehan Li, Zhonghua Wang, Fei Luo, Mingxuan Wang, Xue Yang, Shiwei liu, Yanbiao Ma, Junchi Yan, Jungong Han
Abstract
We introduce LoopVL to study whether Loop Transformers can be effectively extended to vision- language models. LoopVL combines Module-Loop and Model-Loop computation to iteratively update a unified vision-language state through shared modules. We train LoopVL from scratch through language pre-training, multimodal training, and post-training. LoopVL outperforms a range of similarly sized and larger non-recurrent models on multimodal understanding and visual reasoning benchmarks. We also observe Visual Aha Moments in LoopVL, characterized by pronounced shifts in visual attention across loops. LoopVL provides practical evidence for recurrent vision-language modeling and offers an intuitive perspective on how shared parameters can support deeper multimodal computation over continuously evolving visual-language states.