BeyondSight: Object Permanence for End-to-End Autonomous Driving
2026-07-10 • Robotics
Robotics
AI summaryⓘ
The authors address a problem in self-driving cars where other vehicles or obstacles can hide important actors, making the car lose track of them. They created BeyondSight, a system that keeps track of these hidden actors over time instead of forgetting them when they're out of sight. This helps the car better understand and plan around things it can't currently see. They also made a new dataset called nuScenes-Permanence to help train and test such systems. Their experiments show that BeyondSight improves detection and planning accuracy in situations where actors are temporarily hidden.
autonomous drivingocclusionend-to-end drivingactor persistenceobject permanenceprediction and planningnuScenes datasetmAP (mean Average Precision)temporal propagationobservation-conditioned evaluation
Authors
Sandro Papais, Letian Wang, Mudit Jain, Behnaz Rezaei, Steven L. Waslander
Abstract
Autonomous driving operates in partially observable environments where actors may become fully occluded by other vehicles or infrastructure. Most end-to-end driving systems implicitly couple actor existence to instantaneous observations, causing actor hypotheses to degrade or disappear during prolonged occlusion and removing potentially critical agents from downstream prediction and planning. We introduce BeyondSight, a permanence-aware end-to-end driving framework that decouples actor existence from observability by maintaining persistent actor hypotheses over time. BeyondSight propagates actor queries temporally and updates them with observation-conditioned evidence, enabling joint perception, prediction, and planning to reason about actors even when they are temporarily unobservable. To enable principled training and evaluation of persistence-aware models, we further introduce nuScenes-Permanence, an extension of nuScenes that provides supervision and observability-conditioned evaluation for unobservable actors. Experiments show that BeyondSight substantially improves reasoning under occlusion, increasing detection performance for unobservable actors from 0 to 0.249 mAP while reducing planning error from 0.61 to 0.54 L2avg. These results highlight object permanence as an important modeling principle for robust end-to-end autonomous driving.