Prefix tuning controls outputs of pretrained transformers reliably
Prefix-Tuning for Arbitrary Output Sequences on Pretrained Transformers
Computation and LanguageArtificial IntelligenceMachine Learning
Summary
This work answers whether it is always possible to steer a large pretrained language model to produce any desired output by adding a special 'prefix' before the input. The authors prove that if a target output can already be generated by the model, then there exists a prefix that makes the model produce that output every time. They also provide a way to compute this prefix explicitly by solving mathematical equations. Their results explain when prefix-tuning works and when it cannot, and they tested this on models like GPT-2.
What this means in practice
- •For machine learning engineers: Determine precisely when prefix-tuning can produce desired outputs, improving control over deployed transformer models.
- •For software developers for chatbots: Design custom prefixes that reliably generate specific responses in large language models, enhancing chatbot consistency.
Tested on one dataset.
Authors
Peter Cho-Ho Lam, Ziyi Wang, Zirui Zhou
Abstract
Prefix-tuning is a parameter-efficient fine-tuning method for large language models (LLMs) that steers model outputs by prepending task-specific prefix representations to the input sequence. Despite its practical success, the theoretical understanding of the expressiveness of prefixes remains incomplete. In particular, the following question remains open: Given a pretrained autoregressive Transformer, for any input-output sequence pair $(\mathtt{x},\mathtt{y})$, does there always exist a prefix $\mathcal{S}$ such that prepending $\mathcal{S}$ to $\mathtt{x}$ induces the model to generate the target output $\mathtt{y}$? In this work, we prove that under the assumption that $\mathtt{y}$ is in the range of the pretrained Transformer, such a prefix $\mathcal{S}$ exists and can be explicitly computed by solving multiple linear systems. We further show that this assumption is necessary, in the sense that such a prefix may not exist when the assumption fails, thereby providing a complete answer to the above question. Our analysis is constructive and the obtained theoretical results hold under fairly general assumptions on model architecture. Extensive experiments on GPT-2 and Qwen3-4B demonstrate that our algorithm produces prefixes with strong control over model outputs.