Text helps teach image models to spot subtle differences better
Learning Fine-Grained Visual Classification with Text as Privileged Information via Multimodal Distillation
Computer Vision and Pattern Recognition
Summary
Fine-grained visual classification means telling apart things that look very similar by focusing on small details. The authors found that using text descriptions that explain these differences during training helps teach image models to pay attention better. Their method trains a combined model with images and text, then distills that knowledge into an image-only model that works alone during use. This approach improves accuracy and the trustworthiness of the model’s predictions on several fine-grained image recognition tests.
What this means in practice
- •For wildlife monitoring teams: Improve automated species identification by training models with expert text descriptions to better distinguish similar animals in images.
- •For industrial quality control engineers: Enhance defect detection systems by incorporating textual defect attributes during training to refine visual distinction of near-identical product variations.
Authors
Aishwarya Venkataramanan, Christian Ickler, Joachim Denzler
Abstract
Fine-grained visual classification (FGVC) requires distinguishing between highly similar categories by focusing on subtle, localized visual differences. Recent multimodal methods have demonstrated that textual attribute descriptions can provide valuable semantic supervision by explicitly describing discriminative object characteristics. However, these descriptions are typically unavailable at test time, limiting their direct use in deployment. We propose FG-LUPI, a Learning Using Privileged Information (LUPI)-based framework that leverages attribute descriptions during training. We train a multimodal teacher that integrates image patches and textual attributes via gated cross-attention, allowing semantic attributes to condition visual representations. We distill this knowledge into an image-only student using multi-level distillation at the prediction, patch, and attention levels, aligning global predictions, patch representations, and attention patterns with the teacher. This encourages the student to inherit attention structure learned from multimodal supervision while maintaining image-only inference. Experiments on multiple FGVC benchmarks show consistent improvements in classification accuracy over strong ViT-based baselines. Furthermore, evaluation using Expected Calibration Error (ECE) shows that FG-LUPI also produces better-calibrated predictions across all six benchmarks. These results suggest that textual attribute supervision can benefit not only fine-grained recognition performance but also the reliability of model confidence, while preserving the efficiency and practicality of image-only inference.