AdapToPASS: Ambiguity-aware Adaptive Spherical Transformer for Panoramic Semantic Segmentation
Computer Vision and Pattern Recognition
Summary
The authors study methods for understanding 360-degree images, which are tricky because the camera can move and distort the image in unexpected ways. They point out that current models struggle with these distortions because they assume fixed camera positions and don’t adapt well to changes. Inspired by how biological vision adapts to uncertainty, the authors create AdapToPASS, a new model that adjusts its attention depending on local ambiguity and uses special techniques to handle geometric distortions. This model performs better than previous ones, especially when facing new kinds of image transformations, and they also offer a smaller, efficient version called AdapToPASS-Swift.
Spherical TransformersPanoramic Semantic SegmentationSpherical GeometryContextual AmbiguityGeometric AmbiguityAdaptive AttentionBiological VisionBifocal RepresentationBoundary SupervisionModel Robustness
Authors
Soumyaratna Debnath, Weiming Zhang, Shriram Damodaran, Dingwen Xiao, Addison Lin Wang
Abstract
Spherical Transformers have emerged as a promising framework for panoramic semantic segmentation (PASS) by operating directly on spherical geometry and alleviating projection-induced distortions. However, existing architectures often assume canonical spherical structure and stable viewpoints, which are frequently violated in real-world imagery due to unconstrained camera motion, introducing contextual and geometric ambiguity. Consequently, they lack adaptive mechanisms to handle such ambiguity, limiting robustness to unseen spherical transformations. In contrast, biological perception is inherently ambiguity-aware, adapting to fluctuations in cue reliability caused by geometric and contextual variations to maintain stable interpretation under complex transformations. Motivated by this, we first systematically analyze existing PASS architectures under various unseen spherical transformations. We then introduce AdapToPASS, a novel bio-inspired Spherical Transformer that adaptively models contextual and geometric ambiguities for robust PASS. At its core, Adaptive Spherical Attention (AdaSpA) blocks dynamically modulate attention according to local contextual ambiguity, mimicking adaptive, context-driven biological perception. To address geometric ambiguity, AdapToPASS employs Bifocal Spherical Representation to balance field of view and spatial resolution, together with boundary supervision inspired by the boundary-sensitive nature of biological vision. Across indoor and outdoor semantic segmentation, AdapToPASS consistently outperforms prior state-of-the-art methods. Under unseen spherical transformations, it surpasses the next-best method by +13.38% relative mIoU on Stanford2D3D and +18.77% on WildPASS. We further introduce AdapToPASS-Swift, a lightweight variant with fewer than 2M parameters, which surpasses compact baselines while retaining robustness to spherical transformations.