Offline reinforcement learning improves driver interaction in self-driving cars
Beyond Policy Support: Interaction Constrained Offline Reinforcement Learning for Autonomous Driving
Machine LearningArtificial IntelligenceRobotics
Summary
Self-driving cars learn how to drive safely by studying past driving data, but sometimes the system tries actions not well supported by the data, causing mistakes especially around other cars. The authors found that existing methods don't fully consider how a car interacts with surrounding drivers. They created a new method called ICDP that better checks if new driving choices match past interactions, reducing risky decisions. Tests show ICDP helps self-driving systems perform better in tricky driving situations without needing extra simulations.
What this means in practice
- •For autonomous vehicle engineers: Improve decision-making models to select safer driving actions based on past multi-agent traffic interactions without extra simulation.
- •For fleet operation managers: Deploy safer self-driving truck policies that better respect real traffic behavior to reduce collision risks in complex scenarios.
Authors
Mahmoud Selim, Cristina Cipriani, Karl Henrik Johansson
Abstract
Offline reinforcement learning enables reward-driven policy improvement from fixed datasets without requiring online exploration, making it particularly attractive in safety-critical domains. A central challenge, however, is distribution shift: policy optimization may favor actions that are weakly supported by the offline data, rendering value estimates unreliable. Existing approaches primarily control this shift in the policy's own action space. In interactive environments such as autonomous driving, this can be insufficient: a candidate ego trajectory may remain well supported under the marginal behavior distribution while being poorly supported jointly with the surrounding-agent behavior observed in the logged interaction. We refer to this degradation in interaction support as \emph{interaction distribution shift} (IDS), and introduce \emph{Interaction-Constrained Drive Policy} (ICDP), an offline reinforcement learning framework that explicitly controls interaction-level distribution shift. Starting from the joint data distribution over ego and surrounding-agent futures, we show that joint-support degradation decomposes exactly into an ego-support component and a residual interaction-support component. We recover the latter through contrastive density-ratio estimation, isolating interaction compatibility without explicit joint-density modeling, surrounding-agent prediction, or rollouts in reactive simulators or learned world models during policy optimization. Closed-loop evaluations on nuPlan, Interplan and real-world truck experiments show that ICDP suppresses high-value yet interaction-unsupported trajectory selections and improves performance in interaction-critical driving scenarios. Project webpage: https://mahmoud-selim.github.io/ICDP/