On-policy fine-tuning improves camera-based self-driving performance
OPTED: On-Policy Fine-Tuning for End-to-End Driving using a Render-Free Teacher
RoboticsComputer Vision and Pattern RecognitionMachine Learning
Summary
Self-driving cars sometimes make mistakes because their training data doesn’t cover every driving situation. The authors show a method to improve these cars by training a special 'teacher' that knows more detailed map and object data using reinforcement learning. This teacher then helps improve the car’s camera-based driving policy safely and efficiently without heavy simulation. Their method greatly improves driving performance and needs far fewer simulation runs than direct reinforcement learning approaches.
What this means in practice
- •For autonomous vehicle engineers: Improve camera-based driving policies with fewer simulations by leveraging a privileged teacher trained on detailed map and object data.
- •For robotics simulation developers: Use teacher-student setups to reduce computational costs for training sensor-based driving agents in closed-loop environments.
Authors
Damiano Da Col, Maximilian Igl, Peter Karkus, Kashyap Chitta, Boris Ivanovic, Marco Pavone, Konrad Schindler, Christos Sakaridis
Abstract
As scaling pre-training data alone yields diminishing returns, post-training is becoming increasingly important across physical AI domains such as autonomous driving. End-to-end driving policies are pre-trained in open loop with behavior cloning on human demonstrations. However, compounding errors during closed-loop deployment can take the vehicle outside the training data distribution, increasing the risk of safety-critical incidents. Closed-loop post-training can mitigate this risk but requires costly simulation for sensor-based policies. We propose OPTED (on-policy fine-tuning for end-to-end driving) which decouples reinforcement learning from the post-training of the end-to-end policy: a privileged teacher is trained using RL on vectorized inputs (HD-map and bounding boxes). This teacher then provides supervision to the pre-trained student during closed-loop post-training. We apply OPTED to two camera-based models, TransFuser and VaVAM, and fine-tune them in AlpaSim, using neural reconstructions (3DGS) of real driving logs. Driving scores increase by factors of 1.6$\times$ and 9.5$\times$, respectively. In controlled experiments OPTED matches closed-loop performance with approximately three orders of magnitude fewer simulator interactions than direct RL post-training, while staying closer to the human prior. Project page: https://01dami23.github.io/opted/