Hyper-RED improves event camera learning with image semantic graphs
Hyper-RED: Scalable Event Pre-training via Semantic Hypergraph Distillation
Computer Vision and Pattern Recognition
Summary
Event cameras capture data differently from regular cameras, making it hard to teach computers to understand this data well because there isn’t much labeled event camera data available. The authors developed Hyper-RED, a method that links groups of related parts in images and event data instead of matching them one by one. This helps the computer learn the important relationships and meanings from images to better understand event camera data. Tests show this approach works well across several datasets and tasks, improving event camera understanding.
What this means in practice
- •For robotics engineers: Use Hyper-RED to enhance event camera perception modules by transferring rich semantic knowledge from images to event data, improving robustness in visually complex environments.
- •For autonomous vehicle developers: Integrate Hyper-RED pretrained event encoders to improve event sensor understanding without requiring extensive annotated event datasets, boosting real-time vehicle perception.
Authors
Meisen Wang, Zhiqiang Tian, Wei Bao, Chengjie Wang, Shaoyi Du, Siqi Li
Abstract
Event cameras have shown great potential for robust visual perception, yet scaling event representation learning remains challenging due to the scarcity of large-scale annotated event data. Pretrained image models provide scalable semantic supervision, but existing image-to-event methods rely on rigid pixel-wise or token-wise alignment that overlooks modality discrepancies in texture, density, and appearance, potentially causing semantic collapse and limiting transferability. To address this issue, we propose Hyper-RED, a simple, painless, and scalable image-to-event pretraining framework that transfers high-order semantic structures from images to events. Hyper-RED uses hypergraphs to model and align high-order semantic associations among multiple image and event tokens, enabling cross-modal knowledge transfer while accommodating modality-specific differences rather than enforcing rigid one-to-one correspondence. Specifically, given a paired event--image sample, Hyper-RED leverages DINOv3 to extract spatial token representations and constructs image, event, and cross-modal semantic hypergraphs, where each hyperedge connects multiple semantically correlated tokens. We further introduce a hypergraph relational distillation loss that imposes complementary intra- and cross-modal constraints, enabling the event encoder to inherit image-derived semantic organization while preserving local relational consistency and event-specific characteristics. Experiments on three tasks across five event datasets demonstrate consistent scaling from ViT-S to ViT-L and state-of-the-art performance (Fig.1). The code is available at: https://github.com/meisenwang/Hyper--RED.