Large language models improve predictions for automated curriculum learning
Anticipating the Consequences of Curriculum Decisions with Large Language Models
Machine Learning
Summary
Choosing the right order of lessons can help computers learn better, but deciding which lesson to teach next is hard. The authors study how large language models (LLMs) can predict the effects of these curriculum choices more accurately. They combine real-time progress checks with the LLMs’ ability to estimate which tasks will help most and which are ready for training. Testing on hundreds of tasks in a text-based game shows that using LLMs helps especially when focusing on individual goals, with gains depending on how skills transfer between tasks.
What this means in practice
- •For game ai developers: Improve AI training by predicting which game challenges will best advance learning progress using large language models combined with progress signals.
- •For robotics programmers: Design adaptive training sequences that better anticipate the benefits of learning tasks in sequence, speeding up skill acquisition in physical robots.
Authors
Octavio Pappalardo, Nathan Herr, Tim Rocktäschel
Abstract
Automatic curriculum learning can improve the effectiveness of reinforcement learning by selecting the training experiences presented to the agent over time. Predicting the consequences of such decisions can, however, be difficult. We analyze automatic curriculum learning as a sequential decision-making problem, highlighting a gap between the quantities that determine the value of curriculum decisions and the information captured by local learning signals commonly used to guide them. We then investigate whether Large Language Models (LLMs) can exploit richer information about the learning problem to better anticipate the consequences of curriculum decisions. We introduce a method that combines online learning-progress estimates with LLM-informed estimates of (i) the potential downstream benefits of learning on each task and (ii) whether direct training on a task is currently likely to produce progress. We evaluate the approach on a custom benchmark of 256 textual goals in Craftax under different curriculum objectives. We observe the strongest gains when optimizing for individual target tasks. When optimizing across the full task set, the benefits vary across learners with different mechanisms for cross-task transfer, ranging from modest improvements in learning speed to larger gains that persist through the end of training.