Concept bottleneck model accuracy often comes from nonlocal cues
Auditing Where Concept Bottleneck Accuracy Comes From on CUB
Computer Vision and Pattern RecognitionArtificial IntelligenceMachine Learning
Summary
Predictive models that explain their reasoning by naming key features, like a bird’s curved beak, are supposed to be easier for humans to check and correct. The authors found that on a popular bird image dataset, the model’s success often comes not from spotting the actual features but from clues linked to the bird species overall. This means the model’s explanations can be misleading, since they rely on indirect patterns rather than clear, local evidence of the concepts. The paper highlights challenges in evaluating these explanations correctly because the labels themselves can be noisy or inconsistent.
What this means in practice
- •For computer vision engineers: Improve model design by verifying if concept-based explanations rely on true local evidence or nonlocal shortcuts.
- •For quality assurance teams: Develop evaluation protocols that go beyond accuracy and focus on whether model explanations genuinely reflect intended concepts.
Tested on one dataset.
Authors
Sascha Saralajew, Eleonora Poeta, Eliana Pastor, Ammar Shaker
Abstract
Concept Bottleneck Models (CBMs) promise predictions that humans can inspect and correct through concepts such as "curved beak". Yet on CUB-200-2011, their standard benchmark, progress is measured like that of ordinary classifiers: by task and concept accuracy. Neither reveals whether the label is predicted from the exposed concept scores, or whether a concept is supported by visual evidence of that concept rather than by co-occurring features such as cues to the species as a whole. We audit where the accuracy of CBMs and Concept Embedding Models (CEMs) on CUB comes from. The CUB annotations are too noisy to explain it: class-level training labels agree with per-image annotations on only ~82% of image--concept pairs, and classifiers trained on per-image annotations reach only ~37% species accuracy. Leakage probes show that CEM predictions cannot be reconstructed from their concept scores; the missing class information sits in internal embeddings. We identify four design axes through which class information can enter concept scores---receptive field, shared versus concept-specific representations, norm-dependent scoring, and task supervision. An ablation shows that most of it enters through non-local features: restricting a ResNet50 backbone to local BagNet patches lowers the accuracy recoverable from concept scores from 72% to 43%, close to what ground-truth annotations support, while joint training raises it again at the cost of concept accuracy. Localization and occlusion show that locality makes grounding measurable but does not guarantee it: under class-level labels, local concept scores use the head as a proxy for body, wing, and tail concepts, and CEM concept scores barely respond to occluding their supporting part. Task and concept accuracy should thus support, not drive, the evaluation of CBMs, and given the noisy annotations, CUB accuracy gaps should not be overinterpreted.