LP-SFT: Local-Preserving Supervised Fine-Tuning via Multimodal Entropy Structure
2026-07-06 • Computation and Language
Computation and LanguageMachine Learning
AI summaryⓘ
The authors found that when language models are fine-tuned to new tasks, they often lose some of their original abilities because the training only focuses on the correct answer and ignores other good possibilities. They studied the model's predictions and saw that the original model naturally keeps track of several plausible next words, not just one. To fix this, they created a new fine-tuning method called LP-SFT, which tries to keep this variety of options intact while learning the new task. Their experiments show LP-SFT helps the model do better overall without losing its original knowledge or diversity.
Supervised fine-tuningPretrained language modelsCross-entropy lossShannon entropyRenyi entropyNext-token predictionEntropy structureLocal Preservationpass@k accuracyModel capability degradation
Authors
Yueyang Wang, Baolong Bi, Shuo Lu, Jingyuan Zhang
Abstract
Supervised fine-tuning (SFT) is the standard approach for adapting pretrained language models to downstream domains, yet it often improves target-domain behavior at the cost of degrading pre-existing capabilities. Standard cross-entropy fine-tuning promotes only the observed label token and leaves unconstrained how probability mass is redistributed over other plausible alternatives, potentially distorting the rich local preference structure learned during pretraining. We first analyze next-token predictions using Shannon and Renyi entropies, revealing that pretrained models exhibit a regular multimodal entropy structure. These entropy peaks correspond to varying numbers of plausible alternatives, indicating that the base model intrinsically encodes rich distributional knowledge beyond the single supervised token. Motivated by this observation, we propose LP-SFT, a Local-Preserving Supervised Fine-Tuning objective designed to explicitly protect this inherent entropy structure. At each step, LP-SFT constructs an adaptive support of alternative tokens and applies a locally normalized preservation loss to maintain the base model's relative structure among them, while standard cross-entropy independently optimizes the supervised token. Across mixed-domain and single-domain fine-tuning experiments, LP-SFT improves overall performance over vanilla SFT and recent SFT-enhancement baselines, achieving the best balance between pass@1 accuracy and pass@k performance. These results suggest that local preservation helps mitigate capability degradation without collapsing sampling-accessible diversity.