AI summaryⓘ
The authors introduce PixelPilot, a new model for self-driving cars that improves how these systems understand and predict driving paths. Unlike past methods that directly convert 2D camera images into 3D paths—which can be tricky and less adaptable—PixelPilot first plans paths purely within 2D image space, making it easier to train on different datasets. It then converts these 2D plans into 3D paths only when actually driving, helping the model better use visual information to make decisions. Their approach also includes a special learning method to connect what the system sees directly to its planned movements. Tests show PixelPilot works better and adapts more easily than previous models.
Vision-Language-Action ModelsVision-Language ModelsTrajectory Prediction2D-to-3D LiftingSensor-Agnostic PlanningGroup Relative Policy OptimizationCausal Chain in Policy LearningOpen-loop and Closed-loop TestingAutonomous DrivingVisual Reasoning
Authors
Pin Tang, Guoqing Wang, Xiangxuan Ren, Zhongdao Wang, Guodongfang Zhao, Bailan, Chao Ma
Abstract
Vision-Language-Action Models (VLAs), which leverage the advanced reasoning capabilities of Vision-Language Models (VLMs), show promising generalization in complex autonomous driving scenarios. Existing VLAs typically predict and optimize 3D trajectories from 2D images. While intuitive, this 2D-to-3D prediction is inherently entangled with camera parameters, leading to limited data scalability across heterogeneous driving datasets. Moreover, directly optimizing in 3D space induces severe convergence to trivial solutions, where VLAs rely on ego-status rather than visual scene understanding. To address these issues, we propose PixelPilot, a novel VLA featuring a decoupled planning and lifting paradigm. In the planning phase, PixelPilot reformulates scene understanding and trajectory prediction as sensor-agnostic 2D-to-2D tasks in the image plane, thereby facilitating scalable training across diverse datasets. The planned 2D trajectories are then deterministically lifted to 3D only during inference, ensuring the full exploitation of visual cues and generalization across different vehicles. To realize this paradigm, we propose a knowledge-instilled policy learning strategy that applies dense, intermediate rewards via Group Relative Policy Optimization (GRPO) to enforce a rigorous causal chain from visual perception to spatial planning. Extensive experiments demonstrate that PixelPilot achieves state-of-the-art performance in both open-loop and closed-loop settings, validating its superior scalability and visual reasoning capabilities.