Large language models stop learning when updates add little new information
When Updating Stops Being Learning: Rethinking LLM Self-Evolution via learnable information gain
Computation and Language
Summary
Sometimes large language models try to improve themselves by learning from their own generated examples, but this can stop helping after a while and even make performance worse. The authors show that the problem happens because the model keeps seeing similar or redundant data, which doesn't add new knowledge. They propose a way to measure how much new useful information each round of learning provides and use that to adjust training or stop when improvements stop. This method helps avoid the model getting worse over time during self-improvement.
What this means in practice
- •For ai development teams: Control when further training rounds of language models no longer improve quality by detecting low information gain.
- •For nlp system engineers: Use adaptive training regulation to prevent performance degradation during iterative self-training of language models.
Authors
Chenxu Wang, Chaozhuo Li, Xinze Shi, Songyang Liu, Kyrie You Wu, Ziluowen Luo, Shun Zhang, Chenxi Li, Litian Zhang
Abstract
Self-evolution lets large language models (LLMs) improve iteratively using their own generated data, but often suffers from self-evolution degeneration: performance improves, plateaus, then declines. Existing methods address this issue at the component level, targeting either the Questioner or the Solver, and overlook that self-evolution is a tightly coupled system. We propose a holistic framework based on learnable information gain, which measures how much novel, parameterizable information a round provides relative to the previous round. Theoretically, this gain equals the Kullback-Leibler divergence between the two rounds' data distributions plus their entropy change. Practically, it is estimated by fitting a small language model to the previous round and scoring new data via negative log-likelihood. Based on this diagnostic, we propose ATRI (Adaptive Training Regulation via Information-gain), which reweights samples within a round and halts training across rounds when information gain remains low. Experiments on popular datasets demonstrate the superiority of our proposal.