Learning from Mistakes: Rollout-Retrieval Lifelong Policy Learning for Autonomous Driving
2026-06-29 • Robotics
RoboticsArtificial IntelligenceComputer Vision and Pattern RecognitionMachine Learning
AI summaryⓘ
The authors study how self-driving car programs can keep getting better by learning from their own mistakes over time, not just from expert examples. They propose a new method called R²LPL that identifies fixable errors made during driving, figures out the right corrections, and remembers these lessons to improve the policy continuously. Their tests show that this approach helps the driving policy improve steadily and perform better on difficult driving scenarios. This work focuses on using mistakes as useful feedback for ongoing learning in autonomous driving.
Autonomous drivingLifelong learningPolicy learningClosed-loop scenariosExpert demonstrationsCorrective knowledgeRolloutnuPlan benchmarkLong-tail scenariosSupervised learning
Authors
Cheng Gong, Haoyang Wang, Chao Lu, Zirui Li, Jianwei Gong
Abstract
Autonomous driving policies should be able to improve continually as deployment exposes them to increasingly diverse and long-tail traffic situations. However, most learning-based policies are trained or fine-tuned on expert demonstrations and then rely largely on generalization to handle challenging closed-loop scenarios, lacking an explicit mechanism to correct and retain the mistakes exposed in these scenarios. This paper studies autonomous driving policy improvement from a lifelong learning perspective: Can a pretrained policy improve continually by accumulating corrective knowledge derived from its own mistakes, while retaining previously acquired driving competence? To answer this question, we propose Rollout-Retrieval Lifelong Policy Learning (R$^2$LPL), a policy learning framework that retrieves corrective targets from recoverable policy-induced mistakes and retains the resulting knowledge through lifelong policy learning. R^2LPL addresses a key bottleneck in continual policy improvement: closed-loop mistakes reveal where the policy is weak, but do not directly specify what the policy should learn. By filtering recoverable mistake-related states and retrieving feasible corrective targets, R$^2$LPL turns sparse failure evidence into compact supervised knowledge for stable and sample-efficient policy improvement. We evaluate R$^2$LPL on large-scale closed-loop nuPlan benchmarks. With only a few rollout and continual-learning cycles, R$^2$LPL elevates a learning-based planner with moderate initial performance to state-of-the-art performance across the evaluated benchmarks, especially on the challenging and long-tail Test14-hard split. These results demonstrate the effectiveness of R$^2$LPL in converting recoverable closed-loop mistakes into corrective knowledge for sustained policy improvement.