Vision language models reuse transformer blocks for efficiency
Recursive CLIP: Vision-Language Distillation with Looped Transformer Blocks
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Large AI models that look at pictures and read text usually use many separate processing steps. This paper explores if these steps can be reused repeatedly instead of having many different ones, making the model smaller without losing much ability. The authors test this idea by recycling the same transformer blocks multiple times in the model’s vision and text parts. They find that this reuse can work well for some tasks, saving space and making it easier to adjust the model later. This approach offers a new way to make vision-language AI systems more efficient and adaptable.
What this means in practice
- •For mobile app developers: Create lighter vision-language models that use less memory but maintain performance for image and text understanding on phones and tablets.
- •For cloud service engineers: Optimize deployment of vision-language models by reusing transformer blocks to reduce storage and speed up model updates.
Authors
Szymon Mazurek, Alberto Presta, Michał Byra
Abstract
Contrastive vision-language models such as CLIP typically rely on independently parameterized transformer blocks across depth. We investigate the extent to which these blocks can instead be reused recurrently while preserving the transferable capabilities of CLIP. We study recursive transformer backbones in which a small set of $N$ transformer blocks is applied $M$ times per forward pass. By holding $N \cdot M$ fixed to the depth of a standard ViT-B, our models preserve computational depth while progressively increasing parameter sharing. We consider two recursive CLIP model families: rCLIP, where only the vision encoder is recursive, and r$^2$CLIP, where recurrence is also introduced in the text encoder. We study these models under both direct contrastive training and through knowledge distillation from a strong ViT-B/16 teacher pretrained on DFN-2B, using the substantially smaller CC3M and CC12M datasets. We evaluate the resulting models on zero-shot classification and cross-modal retrieval, and further study their adaptability through few-shot learning, visual grounding, and concept unlearning. Our results show that CLIP can tolerate substantial recurrent block reuse, although its effect depends on the encoder and downstream task. Recurrence also provides parameter efficiency by construction, since fewer distinct transformer blocks need to be stored and adapted. Our results establish recurrent block reuse as a practical design space for vision-language models and highlight its implications for knowledge transfer, downstream adaptation, and parameter efficiency.