Prompt update method improves model accuracy on shifted data
Test-Time Prompt-Agnostic Decomposition
Computer Vision and Pattern Recognition
Summary
Large AI models need to adjust when facing new types of data they haven’t seen before. The paper finds that current ways of tweaking prompts during testing can be unstable and sometimes make things worse. The authors introduce a way to break down these prompt updates into parts and control them better to avoid errors from building up. Their method improves accuracy on both visual and text tasks without changing the AI models themselves.
What this means in practice
- •For machine learning engineers: Improve robustness of AI models to new or shifted data distributions by stabilizing prompt updates during test-time adaptation.
- •For natural language processing teams: Enhance text model accuracy on unseen data through controlled prompt update methods that do not require model backbone changes.
Authors
Junze Wang, Lei Fan, Dezheng Zhang, Donglin Di, Yang Song, Sidong Liu, Cong Cong
Abstract
Test-Time Prompt Tuning (TPT) adapts large pretrained models to unseen target distribution shifts by updating only lightweight prompt tokens. Existing TPT methods are effective, but they mainly optimize each prompt update locally. Across test-time steps, prompts updated at the current step are reused for the corresponding predictions and updates, so noisy or biased unlabeled objectives can accumulate and destabilize the prompt-update trajectory. We identify two failure modes: magnitude expansion, where updates move the prompt too far, and directional drift, where consecutive updates point in inconsistent directions. To analyze and control these failures, we propose Test-Time Prompt-Agnostic Decomposition ($\mathtt{TPD}$), which characterizes observed prompt-update trajectories from a dynamical-systems view and decomposes base prompt updates into magnitude and direction. $\mathtt{TPD}$ computes a spectral radius to measure update expansion and decomposes update directions into persistent, oscillatory, and residual components. Building on this decomposition, Adaptive Koopman Control ($\mathtt{AKC}$) regulates update magnitude by shrinking updates under expansive recent dynamics, while Hankel Update Router ($\mathtt{HUR}$) refines update direction by preserving persistent components and suppressing oscillatory and residual components. Together, $\mathtt{AKC}$ and $\mathtt{HUR}$ produce a stabilized prompt update. As a prompt-agnostic framework, $\mathtt{TPD}$ can be plugged into visual and text TPT methods without modifying the backbone, prompt architecture, or adaptation loss. Experiments on 15 datasets show that $\mathtt{TPD}$ consistently improves 10 baselines, with accuracy gains of $3$--$8\%$ for visual prompts and over $2\%$ for text prompts.