Language helps vision models learn better and resist errors
Language Guidance for Supervised Vision Training: An Empirical Study of Generalization
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Vision models usually learn from simple labels that say what an image is but don’t explain how different classes relate. This paper studies how adding guidance from language models can improve these vision models without needing extra paired data or complicated setups. The authors test two ways to add language help—one that adds explicit language info and another that does so more indirectly. They find that these language guides improve how models handle new or tricky situations, resist being fooled by shortcuts or attacks, and remember older tasks better when learning new ones.
What this means in practice
- •For computer vision engineers: Improve image classification models’ robustness to unusual or corrupted images by adding language-based guidance during training.
- •For machine learning operations teams: Maintain performance of vision models when updating them continually by integrating auxiliary language supervision without needing paired datasets.
Authors
Shruthi Gowda, Nikita Jain, Elahe Arani, Bahram Zonooz
Abstract
Deep neural networks have achieved remarkable success on vision benchmarks, yet they continue to struggle with many generalization challenges. Supervised vision training relies on one-hot labels, which provide limited information about semantic structure and shared attributes between classes. This limited supervision can leave visual representations vulnerable to distribution shifts, spurious correlations, texture bias, adversarial perturbations, and forgetting in sequential learning settings. We study whether pretrained language models can serve as lightweight auxiliary supervision for vision training without requiring paired image-text data, prompt engineering, or contrastive objectives. Specifically, we evaluate two forms of language guidance, Explicit Language Guidance (ExLG) and Implicit Language Guidance (ImLG). We conduct a comprehensive evaluation across six generalization regimes, including in-distribution, out-of-distribution generalization, shortcut and spurious correlation resistance, texture and shape bias, adversarial robustness, and continual learning. Our analyses show that the two mechanisms have complementary strengths, with explicit guidance consistently benefiting in-distribution, low-data performance, and continual learning retention, while implicit guidance is often more useful in shortcut-sensitive settings and under stronger adversarial perturbations. Importantly, both are lightweight and add minimal parameters and training overhead. The analyses characterize when language-derived structure helps supervised vision training and provides a practical roadmap for using off-the-shelf pretrained models from another modality as auxiliary supervision.