Texture penalized networks improve image recognition by focusing on shapes
Mitigating Shortcut Learning: Texture-Penalized Prototype Networks
Computer Vision and Pattern Recognition
Summary
Standard image recognition models often make mistakes because they focus too much on tiny textures instead of the overall shape of objects. The authors created a new kind of network called Texture-Penalized Prototype Network that discourages relying on textures and encourages recognizing shapes instead. This approach helps the model avoid confusion when pictures have unusual textures or noise. Their method improves how the model generalizes to new or altered images without needing a lot of extra data. It achieves better shape understanding while only slightly reducing performance on normal images.
Convolutional Neural Networkstexture biasshape biasprototype networkshigh-frequency patternsOut-of-Distribution (OOD)ResNet-50Vision Transformerimage classificationshortcut learning
Authors
Akshay Anilkumar Girija, Elena Hoemann, Frank Köster, Sven Hallerbach
Abstract
Standard Convolutional Neural Networks (CNNs) exhibit severe performance degradation due to a strong inductive texture bias that prioritizes local, high-frequency patterns over global structural shapes. This dependency causes confident misclassifications during textural changes or environmental effects. To address this flaw, this study introduces the Texture-Penalized Prototype Network (TPPN), a novel architectural framework that shifts this inherent bias without depending on resource-intensive augmented datasets. Specifically, a Texture-Penalization Branch (TPB) imposes a penalty to suppress the extraction of local texture proxies, forcing the network backbone to discard high-frequency cues and extract purified, shape-biased representations. By evaluating similarities within a prototype-based hypersphere derived from the final convolutional features, the approach enforces strict geometric constraints, treating objects as compositions of essential parts to achieve robust classification. Evaluations on texture-shape cue-conflict datasets and synthetic noise benchmarks demonstrate the stronger shape bias of this structural disentanglement. The proposed framework reduces the inherent texture bias of a baseline ResNet-50 from 55.11% to 29.73%, surpassing the texture-suppression capabilities of an off-the-shelf Vision Transformer (ViT-B/16). Furthermore, the approach demonstrates robust generalization under cue-conflict conditions, resisting textural shortcut learning when encountering Out-of-Distribution (OOD) shapes. The model maintains stronger shape accuracy against elevated perturbations. On clean validation data, the architecture incurs a minimal drop in accuracy of 0.90 percentage points. This provides a structural, efficient solution to CNN texture bias.