LiFR v2 improves dense perception accuracy and speed using events and RGB
LiFR v2: Completion-Augmented Event Propagation for High-Rate Dense Prediction
Computer Vision and Pattern Recognition
Summary
High-speed cameras often miss details because they take pictures at fixed rates. The authors use event cameras, which capture quick changes but in less detail, combined with regular RGB images to fill in the gaps. Their LiFR v2 system smartly guesses missing information where data is lacking, and remembers past info to improve predictions. This makes object recognition and depth estimation faster and more accurate than before, even when new objects appear suddenly.
What this means in practice
- •For autonomous vehicle teams: Create faster and more accurate perception systems to detect and understand quickly changing road scenes combining event and RGB camera data.$Commercial implications: Enables improved self-driving perception software that processes sensor data beyond traditional camera frame rates for safer navigation.
- •For robotics engineers: Develop real-time dense depth and segmentation models for robots operating in dynamic environments where objects appear or move rapidly.
Authors
Tao Wan, Xiaoshan Wu, Yifei Yu, Bo Wang, Xiaoyang Lyu, Muxin Liu, Aoxuan Pan, Zhongrui Wang, Xiaojuan Qi
Abstract
High-rate dense perception in dynamic environments is limited by the low update rate of RGB cameras, as rapid scene changes can occur between frames. Event cameras offer temporally dense but spatially sparse measurements, complementary to spatially dense RGB observations. Direct fusion cannot fully exploit this complementarity, while event-guided propagation fails on newly appearing or disoccluded regions without valid RGB support. We present LiFR v2, a unified propagation-completion-memory framework for causal anytime and streaming dense prediction from an RGB keyframe and events. LiFR v2 introduces an Event-Guided Completion Module (EGCM) to recover task-relevant representations where propagation is unsupported, and a History Retrieval Module (HRM) to reuse completed representations across successive queries. The framework supports semantic segmentation, monocular depth estimation, and multi-task dense prediction, and we further introduce SHF-Emerge to evaluate rapid object emergence and disocclusion. LiFR v2 achieves 74.37% mIoU on DSEC and 56.13% on SHF-Emerge, improving LiFR-Seg by 1.85 percentage points on the latter, while reducing SHF-Emerge depth RMSE from 1.564 m to 1.118 m over the propagation baseline. It also exceeds 100 FPS for both segmentation and depth, demonstrating accurate and efficient high-rate perception beyond RGB frame rates.