Scaling video prediction improves visual learning for self-driving cars
Scaling Action-Free Video Latent Prediction for Transferable Representations in End-to-End Autonomous Driving
Computer Vision and Pattern Recognition
Summary
Teaching self-driving cars to understand their surroundings is hard because they need to see both the layout and movement over time. The authors looked at using videos without driver actions to teach the car’s vision system before training it to drive. They found that learning from lots of internet driving videos helps the car understand motion and depth better, even when the vision part isn’t retrained later. This way of training works well across different driving methods and tests, showing the approach can generalize.
What this means in practice
- •For autonomous vehicle engineers: Improve self-driving car perception by using large-scale video pretraining without action labels to get better scene and motion understanding.
- •For robotics software developers: Integrate pretrained video encoders frozen from large driving datasets to enhance visual input for different real-time control policies.
Authors
Xingtai Gui, Xubin Zhou, Yucheng Zhou, Dongqian Guo, Jiahao Gong, Feiyang Tan, Jianbing Shen
Abstract
End-to-end autonomous driving policies require effective visual representations that capture both spatial scene structure and temporal dynamics. Video latent prediction has emerged as an effective paradigm for learning spatiotemporal visual representations, yet it remains unclear whether scaling this objective with action-free driving videos can yield representations that benefit end-to-end planning. We study action-free video latent prediction as a scalable pretraining objective and introduce ScaleDrive, a driving-oriented framework pretrained exclusively on large-scale internet open-world driving videos, without action annotations or benchmark-specific data. We keep the pretrained video encoder frozen during downstream end-to-end planning training to isolate the transferability of representations learned from internet driving videos. Ego-motion and depth probes show that scaling action-free video pretraining progressively strengthens the temporal and geometric information encoded in the frozen representation. We systematically study the visual representation transfer across different end-to-end driving policies and closed-loop benchmarks, where ScaleDrive consistently improves downstream performance and demonstrates that scaling action-free video latent prediction yields representations that generalize across both planning policy architectures and evaluation scenarios and protocols.