Summary
Visual systems in AI normally analyze images to understand both what objects are present and their typical colors, even if those colors are not visible. The authors studied how well AI vision encoders remember an object’s usual color when shown grayscale images without any color information. They found that the AI can still identify these typical colors, linking them to the object’s identity, which means the AI holds conceptual color knowledge beyond what is directly seen. Additionally, training vision-language models can significantly change how much color information is retained. This work shows a clear way to test how much AI vision systems understand about object properties beyond the raw visuals.
vision encodervision-language modelscanonical colorconceptual informationgrayscale imagescolor decodabilityobject identitypost-trainingsemantic representation
Authors
Xiaofu Chen, Stella Frank, Yova Kementchedjhieva
Abstract
Visual encoders construct a representation of the image input for Vision-Language models. How much conceptual, as opposed to immediately visible, information does this representation contain? We use canonical color as a controlled test case to ask whether vision encoders make canonical-color information linearly accessible, even when color is removed from the input image. We construct a dataset of objects with canonical colors, and probe vision encoders for both color and object identity using color and grayscale images. We find that canonical color remains decodable from grayscale images, and is tied to predicted object identity, indicating a conceptual link. Extending this analysis to full VLMs, we find that VLM post-training can have a surprisingly large effect on color decodability in the vision encoder. Overall, canonical color provides a usefully controllable lens for tracing object-level conceptual semantic information in vision encoders and VLMs.