Panoptic occupancy benchmark improves 3D driving scene understanding
PanoBench: A Panoptic Occupancy Prediction Benchmark for Autonomous Driving
Computer Vision and Pattern Recognition
Summary
Understanding a 3D driving scene involves knowing where things are, what they are, and which specific object each part belongs to, but this is hard because detailed data is rare. The authors created PanoBench, a new dataset and benchmark that combines synthetic and real-world driving scenes with complete 3D labels including object identities over time. They developed new methods to generate high-quality 3D data from multi-camera views and evaluated different prediction methods on this benchmark. Their work shows improved ways to test and build systems that understand the full 3D environment, including individual objects and background areas, for autonomous driving.
What this means in practice
- •For autonomous vehicle developers: Train and evaluate perception systems that detect and track individual objects and background structures together from cameras in self-driving cars.
- •For robotics simulation engineers: Improve simulation environments by testing algorithms on synthetic scenes with accurate 3D object and background labels matching real-world conditions.
Authors
Yi Feng, Zizhan Guo, Junwu E, Yu Ma, Mengtan Zhang, Jin Wu, Wei Ye, Hanli Wang, Qijun Chen, Danwei Wang, Rui Fan
Abstract
Panoptic occupancy unifies 3D geometry, semantics, and instance identities, but its development is limited by scarce instance-aware annotations. Existing benchmarks mainly provide semantic labels reconstructed from incomplete sensor observations, while panoptic datasets have limited domain coverage or annotation completeness. We introduce PanoBench, a camera-centric panoptic occupancy prediction benchmark spanning synthetic and real-world driving scenes. It combines PanoBench-CarlaOcc, PanoBench-nuScenes, and PanoBench-KITTI360 through canonical data interfaces and unified evaluation protocols, providing multi-resolution annotations with persistent instance identities. The synthetic branch derives complete panoptic geometry from simulated meshes. For real-world data, a new pipeline reconstructs backgrounds using geometric priors and topology constraints, generates foreground mesh assets from multi-frame multi-view images, and fuses them into sparse panoptic voxels. A unified valid-mask protocol evaluates complete thing instances supported by temporal observations while scoring stuff regions at the voxel level. Analyses of observation fidelity, geometric completeness, spatial continuity, and temporal consistency demonstrate the benefits of the resulting annotations over existing occupancy benchmarks. We benchmark representative methods on semantic occupancy prediction, panoptic occupancy prediction, and 4D occupancy forecasting, and study voxel resolution, synthetic-to-real transfer, and storage efficiency. These results reveal complementary strengths of volumetric and ray-based metrics and persistent challenges in real-world instance-aware prediction. PanoBench is available at https://mias.group/PanoBench.