Hybrid vision boosts real-time segmentation with event cameras
Event-Frame Fusion for Inter-Frame Segmentation via Event-Guided Motion
Computer Vision and Pattern Recognition
Summary
Precise understanding of scenes by self-driving cars is hard because normal cameras suffer from motion blur and low frame rates. The authors combined regular cameras with special event cameras that detect quick changes in the scene to improve segmentation accuracy. They used two small neural networks to estimate motion and refine segmentation between frames, enabling faster and more precise results. This approach also helps when regular cameras struggle in bad lighting or motion conditions.
What this means in practice
- •For autonomous vehicle engineers: Enhance perception modules by integrating event-camera-driven motion estimation to improve segmentation under challenging visual conditions.
- •For robotics system developers: Develop real-time navigation systems with higher segmentation speeds and robustness to motion blur using the hybrid event-frame approach.
Authors
Dalia Hareb, Jean Martinet, Benoit Miramond, Elisabetta Chicca
Abstract
Autonomous navigation requires precise and efficient semantic segmentation, yet existing frame-based approaches remain limited by motion blur, glare, latency, and the low temporal resolution (20-30 FPS) of conventional cameras, which leads to information loss between frames. Event cameras have emerged as an alternative sensing modality, capturing intensity changes asynchronously with high temporal resolution, high dynamic range, and sparse outputs. However, event-based algorithms still fall short of frame-based ones in accuracy, as most segmentation methods are designed for dense frame data. To overcome these limitations, we propose a hybrid vision architecture that combines conventional frame-based and event-based cameras. The system integrates two complementary components: (1) a compact Spiking Neural Network (SNN) with 42k parameters for motion estimation, and (2) a lightweight event-driven SNN with 0.84M parameters for frame-based semantic segmentation, which interpolates motion between frames to refine segmentation results. By predicting inter-frame segmentations, the framework achieves segmentation rates of up to 500 Hz with an energy consumption below 1.87 mJ per inference, while maintaining real-time GPU execution at frequencies up to 200 Hz. Additionally, our approach compensates for information loss in frames affected by blur or overexposure, enabling more robust perception in challenging conditions.