Spiking experts improve 3d occupancy resilience in autonomous driving
SpikeOcc: Towards Resilient Multimodal 3D Occupancy Perception via Spiking Experts
Computer Vision and Pattern Recognition
Summary
Reliable detection of objects in three dimensions is important for safe self-driving cars, especially when some sensors fail or go offline. The authors propose a new method using spiking neural networks, which mimic brain-like activity, to build more fault-tolerant perception systems. Their design, called SpikeOcc, separates processing pathways for different sensor types and learns to adapt when some sensors drop out. This approach keeps accuracy high when all sensors work, but also performs better than existing methods when sensors fail.
What this means in practice
- •For autonomous vehicle engineers: Design multimodal perception systems that maintain scene understanding accuracy despite sensor failures using spiking neural network experts.
- •For robotics system developers: Build robust 3D environment mapping systems that adapt to missing or degraded sensor data in real time.
Authors
Yiding Sun, Zijie Xu, Hongqiang Lin, Jiajun Cao, Dongxu Zhang, Kai Tang, Jihua Zhu, Wenxuan Liu, Zhaofei Yu, Tiejun Huang
Abstract
Multimodal 3D occupancy prediction underpins safe autonomous driving through comprehensive scene understanding. However, existing methods have largely pursued higher accuracy through sophisticated fusion, leaving resilience to missing sensors underexplored. In this paper, we investigate spiking neural networks (SNNs) as a means of building more resilient multimodal perception. Still, the discrete nature of spiking neurons makes optimization challenging. To overcome this difficulty, SpikeOcc is introduced with two complementary components, a Spiking Mixture-of-Experts (SMoE) and Sensor Availability Learning (SAL). Specifically, SMoE organizes RGB, LiDAR, and joint fusion into specialized pathways, enabling full-width unimodal representations while retaining the original multimodal features via a bounded residual. Beyond sensor dropout, SAL employs a staged curriculum with explicit routing supervision, progressively improving robustness under severe sensing conditions. Remarkably, SAL also yields notable gains for conventional models, highlighting its broader potential for resilient multimodal learning. Extensive experiments demonstrate that SpikeOcc maintains competitive full-sensor accuracy while markedly raising the performance floor under sensor outages. SpikeOcc illustrates the promise of neuromorphic computing, pointing toward more reliable and energy-efficient autonomous driving.