Spherical geometry improves scene graph creation from 360 degree images
Spherical Geometry-Aware Panoptic Scene Graph Generation for Equirectangular Panoramas
Computer Vision and Pattern Recognition
Summary
Creating detailed maps of things and their relationships inside 360-degree panoramic images is hard because these images are distorted and have seams. The authors developed a method that uses the spherical shape of these images to better combine many smaller views and reason about their relations. They also built a new large dataset of panoramic scenes with detailed labels. Their approach significantly improves accuracy in identifying objects and the connections between them from these challenging panoramic photos.
What this means in practice
- •For robotics developers: Improve scene understanding in robots equipped with 360-degree cameras by reliably combining distorted panoramic views into accurate object and relation maps.
- •For virtual reality content creators: Automatically generate detailed scene graphs from panoramic images to enable interactive environment understanding in VR applications.
Authors
Zhe Wu, Xiaoyan Fei, Xinyi Jiang, Yanxu Wei, Junming Rao, Yihong Yan, Xuewei Li
Abstract
Panoptic scene graph generation (PSGG) represents scene entities with segmentation masks and directed relations. Equirectangular projection (ERP) panoramas challenge PSGG through distortion, latitude-dependent pixel areas, and an artificial seam. We propose a two-stage spherical geometry-aware framework that fuses overlapping perspective-view predictions using observation reliability, spherical overlap, and angular proximity. Relation reasoning combines spherical-area-aware aggregation, weighted no-relation supervision, and dual-longitude score fusion. We also construct a dataset of 340 high-resolution panoramas with 10,571 nodes and 6,632 directed relations. Against corresponding baselines, spherical fusion improves PQ by 36.03% under matched views, while relation reasoning improves mR@50 by 11.20% under PredCls with ground-truth masks and object labels. The full framework improves mR@50 by 52.99% under SGDet with predicted panoptic segments. These results demonstrate the value of spherical geometry for panoramic PSGG.