Selective layer fine-tuning improves large language models accuracy and general skills
Where to Adapt Matters: Layer-Selective Fine-Tuning for Capability Retention
Artificial Intelligence
Summary
Large language models can be fine-tuned to do specific tasks, but this often causes them to forget some of their general abilities. The authors found that fine-tuning only certain layers of the model keeps these general skills better while still improving task performance. They developed a way to identify which layers to adjust by measuring how much the model’s behavior changes at each layer. Their method, called Layer-Selective LoRA, uses this idea to fine-tune only the most suitable layers, improving results on math and coding tasks without losing as much common sense.
What this means in practice
- •For ai model engineers: Develop fine-tuning strategies that preserve a language model’s general reasoning while improving task-specific performance by selecting sensitive layers to train.
- •For software developers using ai models: Integrate layer-selective fine-tuning to deploy fine-tuned models that balance specialized skills and general capabilities for applications like code generation.
Authors
Zhiqiang Pang, Zihong Sun, Qi Xie, Jun Shu, Deyu Meng, Zongben Xu
Abstract
Parameter-efficient fine-tuning (PEFT) enables large language models (LLMs) to adapt to specialized tasks, but often at the cost of degrading general capabilities acquired during pretraining. Existing approaches primarily mitigate this trade-off through data replay or regularization, relying on additional data or explicit optimization constraints. We instead focus on a different question: where should adaptation be applied? We find that fine-tuning different Transformer layers produces different target-task gains and degrees of capability degradation, suggesting that not all layers are equally suitable for adaptation. To characterize this difference, we use layer-wise empirical Fisher information to measure target-task sensitivity. However, computing Fisher scores requires backward computation and becomes increasingly expensive for large models. We therefore introduce input--output cosine similarity as a lightweight, forward-only proxy for ranking layer sensitivity. Across models and tasks, layers with lower input--output similarity consistently exhibit higher empirical Fisher scores. Building on this observation, we propose Layer-Selective LoRA (LS-LoRA), which places trainable LoRA adapters only in layers with low input--output similarity. Experiments on mathematical reasoning and code generation show that LS-LoRA improves average target-task performance while retaining substantially more commonsense reasoning capability than standard all-layer LoRA, demonstrating that carefully choosing where to adapt can provide a simple and effective way to balance target-task adaptation and general capability retention.