Vision encoders retain color info even from grayscale images
Canonical Color as a Lens into Concept Decodability in Vision Encoders and VLMs
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
The paper finds that computer vision systems can recognize the usual colors of objects like bananas or carrots even when shown only black-and-white images. This means these systems have learned conceptual knowledge about objects beyond just the visible features. The researchers built a special dataset to test if 'canonical colors'—the typical colors objects have—are still identifiable in the system’s image understanding. They also showed that training vision-language models changes how this color information is stored in the systems. This helps us peek into how these AI models understand the world beyond what they directly see.
What this means in practice
- •For computer vision developers: Design vision models that maintain object concept understanding even with missing or altered visual features like color.
- •For image recognition system builders: Improve robustness of object recognition systems by using canonical color cues extracted from grayscale or partially occluded images.
Authors
Xiaofu Chen, Stella Frank, Yova Kementchedjhieva
Abstract
Visual encoders construct a representation of the image input for Vision-Language models. How much conceptual, as opposed to immediately visible, information does this representation contain? We use canonical color as a controlled test case to ask whether vision encoders make canonical-color information linearly accessible, even when color is removed from the input image. We construct a dataset of objects with canonical colors, and probe vision encoders for both color and object identity using color and grayscale images. We find that canonical color remains decodable from grayscale images, and is tied to predicted object identity, indicating a conceptual link. Extending this analysis to full VLMs, we find that VLM post-training can have a surprisingly large effect on color decodability in the vision encoder. Overall, canonical color provides a usefully controllable lens for tracing object-level conceptual semantic information in vision encoders and VLMs.