Synthetic data can cause forgetting in large skill retrieval systems
When Synthetic Data Hurts: On Catastrophic Forgetting in Skill Retrieval for LLM Agents
Information RetrievalArtificial IntelligenceMachine Learning
Summary
Selecting the right skills from a huge list is important for AI helpers that use language models. The authors show that training with made-up (synthetic) data helps the AI pick good skills it has seen before but makes it forget how to pick real or new skills. They tested several methods meant to stop this forgetting and found some improved performance across both familiar and new skill choices. This work gives a tested way to train these AI helpers better when there isn’t much real data available.
What this means in practice
- •For llm system developers: Improve runtime skill selection in LLM agents by fine-tuning with methods that reduce forgetting on real and new skill data.
- •For ai platform engineers: Enhance skill routing systems in large repositories by applying continual learning techniques to mix real and synthetic training data.
Authors
Syed Shariyar Murtaza, Yifan Nie, Utkarsh Soni, Eugene Wen, Arvid Frydenlund
Abstract
LLM agents increasingly rely on external skills retrieved at runtime, making skill selection from large repositories a critical challenge. We present a production skill router over 34,396 skills and a large-scale study of skill retrieval using limited real supervision and synthetic data. We found that the synthetic-data fine-tuning improves in-distribution retrieval but it causes catastrophic forgetting on real and out-of-distribution (OOD) data. We evaluate several forgetting mitigation fine-tuning approaches inspired by continual learning, including embedding-anchor regularization, Learning without Forgetting (LwF), Elastic Weight Consolidation (EWC), and L2-initialization. The results show that these approaches not only retain the performance on OOD skills retrieval but also improve the retrieval on synthetic in-distribution skills by 13.98\% for 0.6B Qwen retriever and reranker. Our results provide a practical benchmark and a robust fine-tuning recipe for scarce, multi-positive supervision.