Caption Bottleneck Models
2026-07-01 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors address the challenge of making AI models understandable by replacing fixed sets of predefined concepts with flexible, natural language descriptions generated from images. Their new approach, called Caption Bottleneck Models (CaBM), uses image captions created by large vision-language models to help a classifier make predictions, avoiding shortcut leaks that previous methods struggled with. This method also helps the model discover useful concepts on its own, without needing expert labels or fixed vocabularies. They tested CaBM on different tasks and found it works well while keeping the model easy to interpret.
Concept Bottleneck Modelsinterpretabilitylarge language modelsimage captioningvisual explanationsinformation leakagedataset-specific conceptsfine-grained classificationcoarse-grained classificationtext classifier
Authors
Seref Baris Cagliyan, Umut Ozdemir, Merve Tapli, Emre Akbas
Abstract
Concept Bottleneck Models (CBMs) provide interpretability by routing predictions through a layer of human-understandable concepts. However, defining an optimal concept set for a specific dataset remains an open challenge. Existing approaches rely on expensive expert annotations or LLM-generated lists based solely on class names. Even "open-vocabulary" variants typically depend on static concept sets, which restrict discovery and introduce label bias. Furthermore, traditional CBMs often suffer from information leakage, where unmodeled visual features bypass the bottleneck and compromise the integrity of the explanations. To overcome these limitations, we propose Caption Bottleneck Models (CaBM), a framework that circumvents the need for predefined concept sets by replacing rigid concept layers with free-form natural language. By representing images via LMM-generated captions and training a classifier strictly on this text, CaBM ensures a leakage-free architecture by construction. Additionally, by analyzing the text classifier post-training, CaBM autonomously discovers high-quality, dataset-specific concepts. Our results across fine- and coarse-grained benchmarks demonstrate that CaBM achieves competitive accuracy while preserving interpretability without the constraints of external dictionaries or manual labeling.