Language fields improve street scene understanding with efficient anchor decoding
LangStreet: Persistent Language Fields for Anchor-Decoded Street Gaussians
Computer Vision and Pattern Recognition
Summary
Understanding complex street scenes from different viewpoints is hard because objects look different depending on the camera angle. The authors introduce a new method that organizes scene information into stable 'anchors' and flexible 'slots' to keep track of details as the view changes. Their system efficiently stores and assembles these pieces so it can recognize street objects accurately while using less memory. They tested their approach on popular street driving datasets and achieved similar accuracy to the best methods but with much lower storage requirements.
What this means in practice
- •For autonomous vehicle engineers: Enhance perception systems by efficiently representing street semantics in different camera views to improve object recognition accuracy with reduced memory usage.
- •For urban environment modelers: Create scalable and persistent 3D semantic models of streets that adapt to multiple viewpoints without excessive data storage.
Authors
Runyi Yang, Deheng Zhang, Xiaoye Wang, Mengjiao Ma, Lei Sun, Kanzhi Wu, Ajad Chhatkuli, Luc Van Gool, Danda Pani Paudel
Abstract
Language Gaussian fields implicitly assume that the primitive carrying semantics remains identifiable across views. This assumption breaks in scalable anchor-decoded representations, where persistent anchors generate view-conditioned child Gaussians whose geometry and appearance vary with the camera. We introduce Ours, a persistent language field for such structured Gaussian scenes. Our key idea is semantic ownership: transient children route observations, while persistent decoder slots and their parent anchors own the language field. We use alpha-compositing responsibilities to accumulate additive directional evidence at slots; these statistics marginalize exactly to anchors. We then complete weakly supported slots with anchor-aligned evidence while preserving the anchor direction, and represent slot detail through low-rank residuals in anchor-relative semantic coordinates. Our primary model, Ours (base), stores anchor features together with compact slot residuals. Ours (light) retains only anchor features, whereas Ours (max) stores the full-dimensional completed slot features explicitly. Without scene-specific semantic optimization, Ours (base) nearly matches Ours (max) across KITTI, Virtual KITTI, and Waymo. On KITTI, it achieves 34.19 2D mIoU with a 2.72 GiB effective feature footprint, compared with 34.20 mIoU and 12.90 GiB for Ours (max). The same accuracy-storage trend holds on Virtual KITTI and Waymo. These results show that language fields on view-conditioned splats require persistent semantic ownership, conserved evidence, and a hierarchy that balances stability, detail, and representation cost. Our code, checkpoints, and benchmark suite will be publicly available.