Adaptive curricula improve reinforcement learning by stabilizing training
Training Stability under Adaptive Curricula
Machine LearningArtificial Intelligence
Summary
Training AI agents often involves changing challenges based on their skill level, but this can sometimes cause the AI to stop learning well. The authors found that certain problems in how the AI’s learning behaves cause it to perform worse over time. By identifying and fixing these learning problems, the AI can keep improving and handle new challenges better without needing more training data. This means designing learning tasks and the AI’s training process together is important for success.
What this means in practice
- •For reinforcement learning engineers: Create more reliable AI training methods by stabilizing learning to maintain agent progress under evolving task difficulties.
- •For game development teams: Improve game AI that learns to adapt to player skill by ensuring stable learning despite shifts in training challenges.
Authors
Johan Obando-Ceron, Walter Mayor, Michael Dennis, Aaron Courville, Pablo Samuel Castro
Abstract
Unsupervised Environment Design (UED) approaches train reinforcement learning (RL) agents on diverse curricula that may evolve with their capabilities. While prior work has largely focused on how these curricula are constructed, we study whether their students remain trainable under the resulting distributions. Across multiple UED mechanisms, environment complexities, and extended training horizons, we identify recurring learner-side pathologies, including parameter growth, degraded value--outcome ordering, and loss of representational diversity, that accompany stagnation and performance degradation. These findings connect limitations in UED to optimization failures previously identified in conventional deep RL, while showing how adaptive curricula couple the learner's optimization state to its future training distribution. Controlled interventions show that mitigating selected pathologies improves learning. Stabilizing the learner improves long-horizon performance and zero-shot generalization without changing the curriculum rule or increasing environment interaction. Our results establish learner trainability as a complementary design principle for UED. Effective curricula require students that remain capable of learning from the distributions they produce.