Mapping parameter-efficient fine-tuning methods in neural networks
A Categorical Atlas of Parameter-Efficient Fine-Tuning: Cones, Orbits, and Gauges
Machine Learning
Summary
Fine-tuning large pretrained AI models can be expensive because it often involves changing many parameters. The authors view different fine-tuning methods as mathematical maps from a small parameter space, which helps explain their capabilities and limitations more clearly. Their framework reveals fundamental properties of these methods, such as how they start training, what updates they can represent, and how optimizers behave on them. They also classify hundreds of existing tuning methods using this perspective, proving many exact results about their behavior in training and merging.
What this means in practice
- •For machine learning engineers: Design fine-tuning methods with guaranteed expressivity limits and predictable training dynamics based on mathematical invariants.
- •For ai model integrators: Merge different fine-tuned models precisely while understanding how parameter updates affect weight singular values and training trajectories.
A theory result. No direct application yet.
Authors
Xiaoyu Li, Zhizhou Sha, Chiwun Yang, Dai Shi
Abstract
Parameter-efficient fine-tuning has hundreds of methods, usually compared by parameter count and benchmark score. We compare them as maps. A pretrained checkpoint is a base point in weight space, and a weight-space method is a smooth map from a small parameter space that starts at that point. These maps form a slice category in which the frozen model is initial and full fine-tuning terminal. Three invariants of the map do most of the work. Its image near the base point governs expressivity, through the first-order image, the dimension, a min-cut bound on rank and whether that point is smooth or an apex. Its fibres, with the optimizer's metric, govern training dynamics. Base change moves the base point and covers splitting initialisations, quantisation, restarts and merging. We prove, for example, that zero-initialised LoRA, and Householder reflection adaptation at its default start under stated rank and parity conditions, begin at the apex of a cone; that no exact initialisation lets LoRA reach an update of rank above twice its own; that exactly orthogonal methods never change a weight's singular values in exact arithmetic, however often they are merged; that under Adam with no epsilon, weight decay or clipping and one shared schedule, LoRA+ from the standard zero initialisation follows exactly the trajectory of plain LoRA with its scale multiplied by the learning-rate ratio; that Adam with shared hyperparameters respects only the signed permutations inside LoRA's continuous symmetry group of full-rank factors; and that the activation function decides when a scaling adapter folds into preceding weights. We classify 382 methods from 138 papers by seven coordinates read off their formulas, with 403 simulation arrows and 485 obstructions. Every numbered result is proved in full, and results due to others are credited where we re-derive them.