Language models learn to coach for better problem solving
Learning to Coach for Experiential Learning
Computation and Language
Summary
Large language models (LLMs) can solve problems by learning from past attempts, but their previous answers can be too long and confusing to help much. The authors created a system called Learning to Coach (L2C) where a separate LLM acts like a coach, giving clearer hints based on earlier tries. This coach is trained to guide the original problem solver to improve its next answer and even help with different problems. Tests on math puzzles and text-based games show L2C helps models get better at solving tasks more efficiently.
What this means in practice
- •For ai system developers: Improve AI assistants by training coaching models that guide problem-solving steps more effectively without changing the base solver.
- •For game designers: Create adaptive in-game coaching agents that help players by learning from past player actions to give better real-time hints.
Authors
Guanheng Chen, Tianzhu Ye, Li Dong, Xun Wu, Shaohan Huang, Furu Wei
Abstract
Language models can learn from experience, but raw solution trajectories are often too long and noisy to provide effective guidance. In this work, we propose Learning to Coach (L2C), a framework that trains a dedicated LLM-as-a-Coach to extract actionable experiential knowledge from an actor model's previous trajectory. The actor remains frozen, while the LLM-as-a-Coach is trained to maximize a reward given by the correctness of the actor's guided response. We study two such rewards: a same-instance reward, which improves subsequent responses on the original problem, and a cross-instance reward, which elicits knowledge that transfers to other instances. Across mathematical reasoning and interactive text-games, L2C consistently outperforms self-refinement and an untrained LLM-as-a-Coach. Running experiential learning for more iterations further improves accuracy and uses additional inference compute more effectively than enlarging the actor's decoding budget. The trained LLM-as-a-Coach also transfers to out-of-distribution tasks and adapts its guidance to the specific actor it coaches.