Cartender improves language-guided driving by learning alternatives
Cartender: Verifiable Language-Conditioned Driving by Learning the Drives Not Taken
Robotics
Summary
Driving scenes can lead to many possible paths, but recordings only show the path actually taken, so it’s unclear how well instructions apply to different possible paths. The authors present Cartender, a system that learns not just from the actual driving data but also from alternative, feasible actions that were not taken. It uses a vision-language model to suggest other plausible routes and instructions, then trains a model to judge which instructions fit the scene and which do not. This improves the system’s ability to follow different instructions correctly in the same driving scenario.
What this means in practice
- •For autonomous vehicle developers: Enhance driving policies that interpret natural language commands more reliably by learning from alternative feasible behaviors beyond the logged data.
- •For robot interaction engineers: Improve robots’ ability to follow diverse verbal instructions in complex environments by training on both executed and plausible alternative actions.
Authors
Zhiwen Chen, Benjin Zhu, Yan Xie, Jun Ma
Abstract
Every driving scene is logged once, but driving is one-to-many: the scene determines which behaviors are feasible, while an instruction selects among them. A log records only the action taken, leaving no direct supervision for alternative feasible actions or instructions the scene cannot support. Consequently, a policy that ignores language and one that follows it without regard to scene constraints can fit the same data. We present Cartender, which learns from the drives not taken. Counterfactual Branching constructs matched and mismatched instruction-action pairs from real recordings without a simulator: a vision-language model proposes alternative futures and their selecting instructions, while unsupported instructions provide negative labels. A learned scene-language compatibility score regulates language conditioning in trajectory generation, and a reward-trained language residual re-ranks generated candidates over a scene-based score. The branches are used for training; instruction following is evaluated by swapping instructions under identical recorded inputs and measuring behavior rather than agreement with generated targets. On WOD-E2E, the best of four proposals achieves an RFS of 8.642 across 3,603 feasible instruction conditions. Across 8,974 admitted proposals, mean RFS is 8.203 versus 7.906 for matched logged trajectories, and 67% score higher than their logged counterpart. Compared with matched-only imitation using the same backbone, Cartender improves same-scene paired instruction success from 23.67% to 57.00% and RFS from 6.32 to 7.10; its gate reduces infeasible-instruction execution from 53.36% to 14.18%.