LLM-as-a-Coach: Experiential Learning for Non-Verifiable Tasks
2026-07-20 • Machine Learning
Machine LearningComputation and Language
AI summaryⓘ
The authors show that traditional reinforcement learning methods simplify feedback into single numbers, losing important details. They introduce Experiential Learning, where a language model acts like a coach giving detailed, helpful feedback instead of just scores. This richer feedback helps the model learn better, especially on new and complex tasks. Their approach also avoids problems where models trick their own reward system. Overall, Experiential Learning offers a more effective way to improve language models using complex feedback.
reinforcement learninglarge language modelsrubric-based evaluationscalar rewardcontext distillationexperiential knowledgereward hackingpolicy modelon-policy learningmodel generalization
Authors
Tianzhu Ye, Li Dong, Guanheng Chen, He Zhu, Xun Wu, Shaohan Huang, Furu Wei
Abstract
Reinforcement learning (RL) on open-ended tasks compresses an LLM's rubric-based evaluation into a scalar reward, discarding rich textual feedback and conflating responses with distinct quality profiles. We propose Experiential Learning (EL), which repurposes the feedback model from an LLM-as-a-Judge into an LLM-as-a-Coach. The coach distills its assessment of each on-policy response into transferable experiential knowledge, which conditions a teacher model and is internalized by the policy through on-policy context distillation. Compared with scalar rewards, this higher-bandwidth feedback channel provides dense supervision and preserves fine-grained preferences among high-quality responses. Across two policy families, with feedback from the policy itself or a proprietary model, EL consistently outperforms rubric-based RL on held-out and unseen open-ended tasks. Notably, EL generalizes better beyond the training distribution, and mitigates reward hacking. These findings establish experiential knowledge as a richer and more generalizable learning signal for post-training on non-verifiable tasks.