Visual reprogramming made efficient with one-pass model reading
You Only Reprogram Once: Rethinking Prolonged Training for Visual Reprogramming
Computer Vision and Pattern Recognition
Summary
Training large visual models for new tasks usually takes a long time, even with parts of the model kept fixed. The authors noticed that just observing the existing model’s responses without changing anything can already give good predictions. They created a method called YORO, which uses these frozen responses directly to make predictions without extra training steps. When needed, YORO can refine the input slightly but mostly avoids long training. This approach speeds up adapting visual models while maintaining or improving accuracy.
What this means in practice
- •For computer vision engineers: Quickly adapt large pretrained visual models to new classification tasks using a single forward pass without repeated training.
- •For machine learning platform teams: Reduce computation resources for visual task adaptation by employing YORO's lightweight inference-based mapping instead of costly gradient-based tuning.
Authors
Zizhao Li, Mohammed Yaqoob Ansari, Xinyu Su, Jiayang Ao, Joseph West, Kourosh Khoshelham
Abstract
Visual reprogramming is a parameter-efficient method for adapting pretrained models, yet its training can remain computationally expensive: even with a frozen backbone, visual prompts are often optimized through the full model for hundreds of epochs. Before changing what the pretrained model sees, we ask whether we are fully using what it already tells us. We find that modeling the full source response can already yield strong downstream predictions without prompt optimization. Motivated by this observation, we introduce You Only Reprogram Once (YORO), which constructs a downstream predictor from the frozen response space in a single forward-only traversal. Its Bayesian Discriminant Mapping (BDM) derives a covariance-aware affine mapping from streaming class statistics, requiring no backpropagation, optimizer updates, or repeated visits to the training set. When further input adaptation helps, YORO-FP optionally refines the visual prompt for 20 epochs. BDM also extends naturally to CLIP by treating attribute-prompt similarities as source responses. Across three full-data settings, YORO improves average accuracy over the strongest prior gradient-free mapping by 18.4--24.4\%. On 16-shot CLIP, it raises the four-backbone average from 71.4\% to 77.2\%. YORO-FP provides further gains on selected tasks, while validation often retains the one-pass predictor. These results suggest a different default for visual reprogramming: read out the frozen response first, and optimize the input only when needed.