Large language models keep skills while improving themselves using geometry rules
Mitigating Capability Degradation in LLM Self-Improvement via Representation Geometry Regularization
Artificial Intelligence
Summary
Sometimes large language models get worse at specific tasks as they teach themselves new things. The authors found this happens because the way the model organizes knowledge inside it gets messy. They invented a method called Representation Geometry Regularization that keeps the internal knowledge neat by grouping related ideas tightly and keeping different groups separate. This approach helps the model improve overall while holding on to previous skills.
What this means in practice
- •For machine learning engineers: Maintain robust language model performance during self-improvement cycles in practical deployments by preserving internal representation organization.
- •For ai product teams: Enhance existing automated language model fine-tuning workflows to reduce regression on earlier capabilities without modifying core optimization steps.
Authors
Jiaqi Tian, Ting Long
Abstract
On-Policy Self-distillation is a self-improvement paradigm that enables large language models (LLMs) to improve their capabilities by leveraging the same model as both teacher and student under different conditioning contexts. However, we find that OPSD methods can progressively degrade previously acquired capabilities: queries that are correctly responded at earlier training stages may become incorrectly responded as optimization proceeds, even the overall performance continues to improve. Further analysis reveals that this degradation coincides with a deterioration of the geometry of hidden representations, characterized by reduced intra-cluster compactness and diminished inter-cluster separation. Motivated by this observation, we propose Representation Geometry Regularization (RGR), which preserves representation geometry during post-training by encouraging compactness within cluster and separation across cluster. RGR can be seamlessly integrated into existing OPSD methods without altering their core optimization procedures. Extensive experiments on multiple OPSD methods and benchmarks show that RGR generally alleviates capability degradation while retaining capability improvements. Our code is available in https://github.com/supermachinelearner/RGR.