What's Hidden Matters: Identifying Planning-Critical Occluded Agents using Vision-Language Models
2026-07-01 • Robotics
RoboticsArtificial IntelligenceComputer Vision and Pattern Recognition
AI summaryⓘ
The authors address the problem of self-driving cars needing to deal with hidden or occluded objects that might affect their path. They created a way to rank these hidden objects based on how much they could change the vehicle's planned route using an information measure called Planning KL-divergence. Then, they used powerful language models to better understand and describe these important hidden agents. Testing on a driving dataset showed that training models with this focused data helped smaller models outperform much bigger ones when identifying critical hidden obstacles. This work helps autonomous vehicles better predict and react to hidden risks in a smart and efficient way.
Autonomous vehiclesOcclusionPlanning KL-divergenceVision-Language ModelsGPT-5nuScenes datasetRisk assessmentFine-tuningTrajectory planningInformation-theoretic metric
Authors
Amirhosein Chahe, Tyler Naes, Jovin D'sa, Faizan M. Tariq, Sangjae Bae, Lifeng Zhou, David Isele
Abstract
Autonomous vehicles must safely navigate complex environments where planning-critical agents may be hidden from view. Current approaches often treat all occlusions with uniform conservatism, yielding needlessly defensive driving, or they infer hidden spaces without estimating the impact on the planner. This work bridges the critical gap between perception and planning by enabling Vision-Language Models (VLMs) to identify and reason about the specific hidden agents that are most critical to the ego-vehicle's trajectory. We introduce a novel framework that uses Planning KL-divergence (PKL), an information-theoretic metric, to systematically identify and rank occluded agents based on their impact on the ego vehicle's plan. Using this planning-aware ranking, we employ an expert VLM (GPT-5) to generate rich, structured annotations that capture the visual evidence and reasoning required for this task. We apply this framework to the nuScenes dataset to create a new benchmark focused on high-impact scenarios. We conduct comprehensive experiments on a wide range of general-purpose and domain-adapted VLMs, demonstrating that fine-tuning on our PKL-guided data yields dramatic performance improvements across all models. Notably, our results show that smaller, fine-tuned models significantly outperform their much larger zero-shot counterparts, and that our PKL-guided data selection strategy improves performance by approximately 30\% over random sampling. Our work presents the first systematic approach for training VLMs to focus on planning-critical occlusions, enabling more semantically grounded and efficient risk assessment in autonomous driving.