Self modifying visual backbones improve image recognition and segmentation
VisionHOPE: Visual Backbones as Self-Modifying Learning Systems
Computer Vision and Pattern Recognition
Summary
Image recognition systems usually process pictures using fixed steps learned beforehand, but this paper introduces VisionHOPE, a system that changes how it learns and remembers while looking at an image. VisionHOPE uses several memories that work together to adapt its thinking in real time as it scans an image. To keep this constant change stable and reliable, the authors developed a careful way to control learning updates. Their method performs well on standard visual tasks like identifying objects and segmenting images, showing that self-modifying learning can be practical for visual AI.
What this means in practice
- •For computer vision teams: Integrate VisionHOPE into image analysis pipelines to improve adaptability and accuracy in tasks like object detection and segmentation.
- •For autonomous vehicle developers: Use VisionHOPE backbones to enhance real-time scene understanding by allowing models to adapt continuously to changing visual input.
Authors
Siran Peng, Tianshuo Zhang, Tianyu Fu, Weisong Zhao, Haoyuan Zhang, Jiankuo Zhao, Minghui Wu, Ping Jiang, Xiangyu Zhu, Chenxu Zhao, Zhen Lei
Abstract
Visual backbones have evolved from Convolutional Neural Networks (CNNs) with local aggregation to Vision Transformers (ViTs) with global interactions, State-Space Models (SSMs) with input-dependent state transitions, and Test-Time Training (TTT) layers that adapt an inner learner while processing an image. Across this progression, visual computation has become increasingly adaptive to each input, yet the rules governing that adaptation remain largely prescribed by the trained backbone. We introduce VisionHOPE, the first generic visual backbone formulated as a self-modifying learning system, in which what the model remembers and how it learns co-evolve within an image. Building on the self-referential construction of Nested Learning (NL), VisionHOPE realizes this co-evolution through five coupled memories that store content, generate key and value representations, and govern learning rate and retention. These memories evolve jointly as visual context accumulates along each scan. However, directly applying the unconstrained self-referential update to a visual backbone leads to instability. We therefore derive a stability-matched step-size control scheme that combines a soft cap on self-referential injection with a spectral clamp on the retained memory transition, and prove that the resulting memory dynamics are non-expansive along each scan. For two-dimensional feature maps, we adapt NL's chunk formulation by aligning chunks with image rows and columns across four directional scans. The proposed VisionHOPE achieves competitive results on ImageNet-1K, COCO, and ADE20K, establishing self-modifying learning systems as a practical foundation for general-purpose visual backbones. The code is available at https://github.com/PSRben/VisionHOPE.