LoRA fine-tuning saves resources by focusing on task key layers
Mechanistic Insights into LoRA: Layer Sparsity for Adaptive Fine-Tuning via Path Patching
Machine LearningComputation and Language
Summary
Fine-tuning big language models usually changes all parts of the model, which can be inefficient. The authors find that only some layers really change a lot, and which layers matter depends on the task. They design a method called Adaptive-LoRA that picks just the task-important layers to fine-tune automatically. This method uses fewer resources but works better than fine-tuning all layers equally. Their experiments show it can reduce parameters by over three times while improving performance on multiple reasoning tasks.
What this means in practice
- •For ai model engineers: Reduce compute and memory by fine-tuning only key layers identified automatically for each task in large language models.
- •For cloud ai service teams: Deploy smaller adapters tuned per task to speed up inference and reduce storage while maintaining performance in language AI services.
Authors
Fei Zuo, Yizhou Huang, Kezhi Wang
Abstract
Low-Rank Adaptation (LoRA) has emerged as the dominant paradigm for parameter-efficient fine-tuning of large language models. Standard all-layer LoRA applies adaptation throughout the transformer, despite heterogeneous contributions from its components. In this work, we conduct a systematic mechanistic interpretability analysis to understand how LoRA affects model internals. We observe that LoRA's adaptation effect is highly non-uniform, with significant component-wise and layer-wise sparsity. Most critically, we find that important layers are highly task-dependent, with limited overlap across tasks. Motivated by these insights, we propose Adaptive-LoRA, a calibration-driven method that automatically identifies task-specific layers via frozen-base zero-ablation, enabling efficient layer selection without exhaustive search. Extensive experiments on Qwen2.5-1.5B across 11 diverse reasoning tasks demonstrate that Adaptive-LoRA outperforms full-layer LoRA while using only 30% of layers, achieving a $3.5\times$ reduction in adapter parameters. This work bridges mechanistic interpretability and practical efficiency, providing both fundamental insights and a principled approach for adaptive fine-tuning.