Language models improve test-time learning with group feedback
Continual Test-Time Learning with Evolution Strategies
Machine LearningComputation and Language
Summary
Many AI language models need to keep learning when tackling new or changing tasks, but they often lack outside help or labeled examples to learn from. The authors of this paper created a method called ESRL that uses a group of slightly different versions of the AI to generate teaching signals for the main model, instead of just relying on the model itself. This group approach helps the model learn better without getting lost or forgetting previous knowledge. Their experiments showed ESRL improved accuracy across math, general knowledge, and medical tests while reducing forgetting.
What this means in practice
- •For ai system developers: Build adaptive language models that improve over time during deployment without needing extra labeled data or supervision.
- •For automated customer support teams: Deploy chatbots that maintain and enhance accuracy in real-time as user queries evolve without manual retraining.$Commercial implications: Enables sale of chatbots that self-improve continuously, reducing support costs and improving user experience.
Authors
Guangya Hao, Yiting Duan, Longyan Tan, Yunbo Long, Hao Wu, Ge Zheng, Alexandra Brintrup, Zhuokai Zhao
Abstract
Large language models increasingly operate in open-ended environments, where they are expected to continually adapt to evolving tasks and distributions, often without access to external supervision. Existing continual learning approaches rely on external supervision, such as labeled data, demonstrations, or reward signals, which can be costly or unavailable in practice. Test-time adaptation removes this dependence by learning from self-generated supervision, but existing methods derive that supervision from the single policy being updated, so once the policy drifts, its own learning signal drifts with it, leading to policy collapse, degraded reasoning, and catastrophic forgetting. In this paper, we propose \textbf{Evolution Strategy Reinforcement Learning (ESRL)}, a continual test-time learning framework that draws its self-generated supervision from a population of policies instead of the single policy being trained. ESRL perturbs an anchor policy in parameter space to form a population of nearby policies, samples one trajectory from each, and aggregates their answers into label-free group-relative learning signals for the central student. The anchor is held fixed within each block and refreshed from the student at block boundaries, so the population stays centered on a stable yet current policy and continues to supply diverse and reliable supervision as the student moves. In single-task test-time adaptation across mathematical reasoning, general knowledge, and medical domains, ESRL improves average accuracy by up to 31.7% relative to the base model and 16.6% relative to the strongest test-time adaptation baseline. Notably, ESRL substantially reduces catastrophic forgetting and achieves the strongest final performance in continual learning.