When fixed prefixes can match frozen low rank adapters in transformer attention
When Can Prefixes Compile LoRA? Exact Resource-Capped Tests for Frozen Attention
Machine Learning
Summary
This paper studies whether a fixed continuous prefix can replace a special type of adapter in transformer attention without changing the attention itself. The authors show that this depends on certain mathematical conditions related to how the adapter affects the attention computations. They develop exact tests that examine when such replacements are possible and find limitations related to numerical precision and the type of queries involved. Their analysis focuses on small parts of the model (one attention head) rather than the entire network.
What this means in practice
- •For machine learning engineers: Verify when prefixed inputs can exactly reproduce low-rank adapter effects in frozen transformer heads to optimize model tuning workflows.
- •For ai hardware developers: Design numerical precision requirements for transformer accelerators by understanding limitations in prefix-based approximations of adapters.
Authors
Joyanta Jyoti Mondal, Ibne Farabi Shihab
Abstract
Can a fixed continuous prefix replace a given low-rank adapter while the attention head stays frozen? In this research, we show that the answer depends on the adapter's target through three conditions. First, observability: at one causal readout, every independent key--value prefix sees the content only through the query, attention partition, and value numerator, so a target that differs on two inputs with equal summaries incurs an error floor at every prefix length; norm caps extend this floor to nearly equal summaries. Second, realizability: at a common query, any prefix reduces exactly to two aggregate variables, and the norm-capped optimum is an attained second-order-cone program, also after a fixed output projection; it places two equal-norm rank-one value updates on opposite sides of compilability. Third, implementation: under affine query exposure, $2r$ signed slots approximate a rank-$r$ value update, but their values grow as $O(ε^{-3/2})$, and the construction passes all 400 tolerance checks in float64 yet only 38 in bfloat16. A first-layer GPT-2 readout with fixed token and position meets the common-query condition without clamping activations; at three such heads, the capped optimum leaves 18.4\% to 74.2\% of the projected adapter effect uncompiled, with a head-dependent value--query ordering. All claims concern local approximation at one head, not whole-network equivalence.