Aligning Multi-Trajectory Supervision with Policy Optimization for VLA Driving
2026-08-31 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial IntelligenceMachine Learning
AI summaryⓘ
The authors study how to improve driving models that learn from multiple possible driving paths (trajectories) and then optimize their decision-making policies. They found that some good-looking trajectories can actually mislead the policy update by suggesting unsafe or unrealistic actions. To fix this, they propose filtering these trajectories with a special rule (Pareto-optimality) and keep only those that are both feasible and helpful. They also introduce methods to guide the learning process toward safer behaviors and update the teaching examples over time. Their approach improves driving performance and reduces failures on a driving simulation benchmark compared to previous methods.
Vision-language-action drivingMulti-trajectory imitation learningGroup-relative policy optimizationTrajectory selectionPolicy gradient biasPareto-optimalityFeasibilityAdvantage estimationDynamic distillationSafe behavior
Authors
Tian Zhang, Zhuo Huang, Hongrui Ye, Yu Wu, Zengmao Wang, Kaixuan Zhou
Abstract
Vision-language-action (VLA) driving methods increasingly combine multi-trajectory imitation learning with group-relative policy optimization (GRPO), making trajectory selection critical to final performance. However, some high-scoring trajectories that improve imitation can degrade subsequent GRPO by inducing advantage estimates misaligned with the current policy's feasible behavior distribution, driving updates away from safe and compliant behaviors. To address this, we propose a novel framework that aligns multi-trajectory supervision with policy optimization. To address the policy gradient bias induced by infeasible noisy trajectories outside the feasible region, augmented trajectories are constrained to a neighboring manifold of the ground-truth feasible region, and a Pareto-optimality criterion is adopted in place of the conventional aggregate score, retaining only non-dominated candidates and thereby filtering out conflicting samples at the source. To ensure that expanded trajectory supervision is effectively absorbed during policy optimization, we introduce two complementary mechanisms: feasibility-first advantage assignment and dynamic distillation. The former adapts Pareto credit to the feasibility composition of each rollout group and guides fully infeasible groups toward safe references. The latter updates teacher trajectories across refinement rounds to continually transfer useful supervision. Together, they progressively translate the benefits of expanded supervision into policy improvement. On NAVSIM v1 and v2, our method achieves 91.4 PDMS and 89.1 EPDMS, respectively, under single-trajectory inference, and recovers 440 of 658 initially failed scenes, 11.1\% higher than the original GRPO baseline.