Color affects how humans and AI recognize partly hidden objects

Impact of color in computer and human-based object recognition under occlusion

Computer Vision and Pattern Recognition

Summary

Recognizing objects when they are partly covered is hard for both people and computers. This paper looks at how color influences this challenge, especially when the covering blends in like camouflage. The authors compare human recognition with different types of AI models and find that color camouflage affects them differently. For example, human and transformer models struggle more with camouflage that blends into the scene, while some CNNs do better when color matches the background. Training AI with camouflaged objects can help some models recognize hidden objects better.

What this means in practice

  • For computer vision engineers: Improve robustness of object recognition models in cluttered or camouflaged scenes by augmenting training data with color-based occlusions.
  • For robotic perception teams: Design object detection systems that account for color camouflage effects to maintain accuracy in complex, partially occluded environments.

Authors

Courtney M. King, Elissa Aminoff, Daniel D. Leeds

Abstract

Partial occlusions challenge object recognition in both human and computer vision (CV), but the impact is largely dependent on the occlusion and model design. Properties relating to color are particularly important in biological recognition and frequently leveraged for camouflage, however there are still open questions about what makes them effective and how they translate to CV. This work studies partial occlusions that create camouflage, including those that blend in with target objects or with scenes, which alter object representations in distinct ways. Experiments evaluate recognition performance on both human participants and models ranging from convolutional neural networks (CNNs) to transformers, and additional analysis is performed on models to uncover trends in processing and object representations under multiple occlusion types. Novel metrics offer explainability of model behavior under occlusion: non-occlusion precision (NOP) and non-occlusion recall (NOR) measure alignment of model activations with and without occlusion, and activation in occlusion (AIO) measures the percentage of activations that occur in occluded regions. Findings show that human recognition, and most transformers, are negatively impacted by generated camouflaging occluders, with scene-based camouflage performing worse than object-based. Classical CNNs show a strong opposite trend, performing the worst on neutral-colored occlusions, suggesting that contextually-related colors, particularly scene-based, benefit object recognition rather than disrupting shape or boundary cues. Additionally, generated camouflaging occluders for training augmentation improves robustness to unseen camouflage for selected CNN model, and occlusion metrics suggest trends about learned mechanisms. Findings have interesting implications for alignment in human and computer vision, model vulnerabilities, and explainability.