Fine-tuning language models to balance learning and generalization
SoFT: Soft Targets for Generalizable LLM Fine-Tuning
Machine LearningArtificial Intelligence
Summary
Training smaller language models to learn from multiple expert teachers can make them better at different tasks, but it’s hard to keep them good at things they weren’t explicitly trained on. The authors study how different fine-tuning methods affect the balance between learning new skills and staying versatile. They propose a new fine-tuning method called SoFT that gently nudges the model to learn new tasks while keeping old abilities intact. Tests show SoFT improves performance both on tasks the model was trained on and on new, unseen tasks.
What this means in practice
- •For machine learning engineers: Train smaller language models that maintain versatility across diverse tasks using multi-teacher data from various domains.
- •For ai platform developers: Improve language model fine-tuning pipelines with adaptive controls to balance task-specific learning and generalization.
Authors
Huihao Jing, Wenbin Hu, Shaojin Chen, Haochen Shi, Zhongwei Xie, Guijia Zhang, Yuxuan Liu, Haoyu Huang, Haoran Li, Yangqiu Song
Abstract
Distillation enables student language models to acquire new capabilities from expert teachers. However, integrating knowledge from multi-teacher, multi-domain demonstrations into a single student remains challenging. We study supervised fine-tuning (SFT) in this setting, where students must acquire diverse capabilities while maintaining generalization beyond the training tasks. Our experiments reveal varying trade-offs between in-distribution learning and out-of-distribution generalization across SFT methods, motivating more explicit control over this balance. To this end, we propose soft-target fine-tuning (SoFT) to balance learning from teacher demonstrations with retaining the Base model's existing capabilities. SoFT sets a minimum target probability for each demonstrated token while making the smallest KL change to the Base distribution. The resulting objective couples learning from demonstrations with adaptively weighted regularization toward the Base model. We further use domain-specific gradient budgets to control this balance and determine a probability threshold for each trajectory. Experiments on mixed-domain reasoning and agentic tasks show that SoFT achieves the best overall performance among the compared methods, with improvements in both in-distribution capability acquisition and out-of-distribution generalization.