New method balances learning new skills and keeping old AI abilities
PG-SFT: Balancing Capability Acquisition and Retention in Offline Agent Fine-Tuning
Artificial Intelligence
Summary
When training AI agents by showing them examples of how to do specific tasks, the AI can get better at those tasks but lose some of its other useful skills like general reasoning or code generation. The authors studied how to better keep the AI’s original abilities while still learning new tasks. They found that a method called Privilege-Guided SFT uses information about how valuable each part of a training example is to adjust learning strength, helping the AI keep more of its old skills while improving on new ones. This approach slightly reduces performance on the new task but better preserves broad AI abilities.
What this means in practice
- •For ai application developers: Train AI agents that acquire new task-specific skills without losing general abilities like reasoning or code generation.
- •For software engineers maintaining legacy ai: Update deployed AI models with new tool-using behaviors while preserving existing capabilities to avoid performance regressions in production.
Authors
Ronghua Li, Zi Liang, Zhishan Li, Shinan Liu
Abstract
Supervised fine-tuning (SFT) on offline agent trajectories is the standard approach for training specialized tool-using agents, but forcing models to imitate reasoning and actions token by token may harm other capabilities (e.g., general reasoning, tool calling, code generation) of the base model. In this work, we focus on studying \emph{how to better balance the trade-off between acquiring new capabilities and preserving existing ones during agent trace SFT}. By comparing several baselines in our setup, standard SFT improves the target benchmark while lowering several non-target benchmark scores; meanwhile, simply constraining distributional drift using KL penalty or limiting the update magnitude did not avoid this regression trend. Motivated by recent token-wise adaptive learning objectives, this work proposes \textbf{Privilege-Guided SFT (PG-SFT)} to leverage turn-level information gain of agent trajectories as an indicator to adjust supervision strength. PG-SFT yields a more favorable observed trade-off on the evaluated benchmarks, substantially reducing distributional drift and broad capability degradation at the cost of slight degradation in target-task performance. Our findings suggest that balancing the acquisition--retention trade-off depends not only on whether the model is anchored to its base behavior, but also on where and how strongly supervision should depart from that behavior.}