Method transfers task tweaks across different transformer models
From Parameter Rebasin to Functional Transfer Across Heterogeneous Transformers
Machine Learning
Summary
Fine-tuning large language models for specific tasks creates unique changes that usually cannot be reused across different versions of these models. The authors introduce Ariadne, a method that transfers learning effects by focusing on how activations change, rather than moving parameters directly. Ariadne aligns parts of one model with another and generates updates that work well on new, different transformer models without retraining. This approach saves time and memory and helps apply learned improvements across transformers that vary in size or architecture.
What this means in practice
- •For machine learning engineers: Create task-specific updates that adapt pretrained transformers of varying sizes without retraining from scratch.
- •For cloud ai service operators: Efficiently deploy fine-tuned transformer models at multiple scales using fewer resources by transferring learned behaviors directly.
Authors
Filippo Rinaldi, Aniello Panariello, Giacomo Salici, Simone Magistri, Angelo Porrello, Andrew D. Bagdanov, Simone Calderara
Abstract
Adapting pretrained Transformers to downstream tasks produces task-specific updates that are expensive to reproduce for every model variant. Reusing these updates across different depths and widths is challenging because their parameters have incompatible shapes and do not share a canonical coordinate system. We introduce Ariadne, a training-free method that transfers task-specific behavior across heterogeneous Transformers without resizing the source model or transporting its parameters. Rather than treating the source task vector, i.e. the difference in parameter space between the fine-tuned and the base models, as the object to move, Ariadne characterizes fine-tuning by the activation changes it induces at matched blocks. First, it aligns these effects to a target base through local Procrustes maps. Then, it directly synthesizes a native target update by closed-form ridge regression. Across Transformer expansion, reduction and language-model transfer, Ariadne improves target-model performance while substantially reducing the time and memory required to construct an update relative to previous cross-architecture rebasin approaches. These results show that local functional effects provide an efficient alternative to parameter transport.