Event camera system improves privacy aware emotion recognition
Emo-DVS: A Multimodal Benchmark for Privacy-Aware Emotion Recognition with Event Cameras
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Emotion recognition using regular cameras risks exposing people's private facial details. The authors use special event cameras that capture only changes in light, which helps keep identities private while still detecting emotions. They created a new large dataset with event camera data plus audio and text to make emotion recognition better. They also developed a method to combine these different types of information effectively, improving emotion detection in real-life situations.
What this means in practice
- •For security system developers: Enhance emotion recognition in surveillance systems while reducing privacy risks by using event cameras with multimodal data fusion.$Commercial implications: Enables new privacy-aware emotion detection products for security and monitoring markets using event camera hardware and audio/text signals.
- •For human computer interaction designers: Build user interfaces that adapt to subtle emotional cues captured by event cameras combined with audio and text inputs.
Authors
Jiaqi Chen, Qinfu Xu, Hao Zhuang, Liyuan Pan
Abstract
Emotion analysis is a fundamental task in computer vision, but its practical deployment remains constrained by the privacy risks inherent to conventional RGB cameras. Bio-inspired event cameras present a promising hardware-level solution because they capture asynchronous brightness changes, thereby reducing exposure of facial identity details while leveraging high dynamic range for robust perception under challenging illumination conditions. Despite these advantages, existing event-based methods struggle in complex real-world settings due to limited dataset scales, simple acquisition conditions, and reliance on single-modality visual cues. To address these, we establish a challenging tri-modal benchmark with event, audio, and text modalities and propose the Information-Guided Gated Fusion (IGF) framework, which first pre-trains an event encoder on the FAU subset of Emo-DVS to capture fine-grained facial dynamics, then employs adaptive modality gating to suppress modality-specific noise, and finally leverages mutual information maximization to align robust cross-modal representations. To alleviate data scarcity, we introduce Emo-DVS, the first large-scale event-based emotion analysis dataset, which couples dynamic illumination with the Facial Action Unit (FAU) subset and emotion subset. Extensive experiments demonstrate that IGF achieves state-of-the-art performance.