Language-Critique Imitation Learning from Suboptimal Demonstrations
2026-07-01 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors address the problem of teaching computers to imitate behavior from imperfect examples, which usually rely on simple numbers to judge performance. Instead, they use natural language descriptions to explain what is going well, what is wrong, and how to fix mistakes during learning. They created a way to train computer programs using these detailed language cues without reducing them to just numbers, and showed their method works better than existing ones in tasks like navigation and playing games. They also provide a theoretical guarantee that their approach improves learning compared to previous methods.
imitation learningsuboptimal demonstrationsnatural language supervisionbehavior cloningdiffusion policieslanguage-critique losscontinuous control tasksoffline reinforcement learningpolicy learningstructured feedback
Authors
Chih-Han Yang, Dai-Jie Wu, Yun-Ping Huang, Ping-Chun Hsieh, Kenneth Marino, Shao-Hua Sun
Abstract
Prior work on imitation learning from suboptimal demonstrations typically relies on compressed supervision signals such as confidence estimates, discriminator scores, or importance weights. These scalar signals are inherently limited, as they cannot explicitly express intermediate reasoning about task progress, failure modes, or corrective actions. We propose a language-critique framework for imitation learning from suboptimal demonstrations that instead leverages natural language as a structured supervision signal, avoiding the collapse of expressive feedback into scalars. Our method first constructs language labels from demonstrations that explicitly describe current progress, identify suboptimal behaviors, and provide fine-grained corrective guidance. We then introduce a language-critique loss that directly trains policies using these structured signals without reducing them to scalars, and instantiate it for both behavior cloning and diffusion policies, yielding LC-BC and LC-DP. We further provide a theoretical result showing that the proposed objective upper-bounds the expert performance gap under standard assumptions. Empirically, we evaluate on diverse continuous control tasks spanning navigation, manipulation, and gameplay, where our methods consistently outperform strong imitation learning and offline reinforcement learning baselines. These results demonstrate that language can serve as a powerful and structured form of supervision for learning robust policies from suboptimal data.