Decoding dynamic video perception and recall from brain signals
Replaying the Movie in Your Mind Decoding Dynamic Visual Perception and Recall from EEG
Computer Vision and Pattern Recognition
Summary
It is difficult to know what parts of what you see are remembered in your brain. The authors created a new brainwave (EEG) data set from people watching short video clips and then immediately remembering them. They compared brain activity when people saw videos and when they mentally replayed them, using computer models to try to tell the difference. The results show it’s possible to decode details about what’s seen and recalled, and that the brain signals for seeing and remembering are distinguishable. This work helps us understand how memories of visual scenes form and differ from actual perception.
What this means in practice
- •For brain computer interface developers: Improve decoding of visual memories from EEG to enhance communication tools for people who cannot speak or move.
- •For neurofeedback practitioners: Develop training methods that detect and differentiate visual perception versus recall states using EEG in real time.
Authors
Zikai Fang, Tian-Yi Zhou, Xuan-Hao Liu, Yixi Li, Xiaoyu Xing, Yutong Pei, Bao-Liang Lu, Wei-Long Zheng
Abstract
Decoding dynamic visual content from EEG has advanced rapidly for visual perception, yet it remains unclear which aspects of neural representations are preserved when people internally replay dynamic scenes. We introduce PAIR, an EEG benchmark collected from 20 participants using an interleaved, clip-wise paradigm that pairs naturalistic video viewing with immediate visual recall. This design enables a comparison between externally driven perception and internally generated recall for the same stimulus content within a fixed trial sequence. We benchmark multiple decoding models on hierarchical semantic classification at both fine-grained and coarse-grained levels, meta-feature decoding tasks covering motion, color, object number, face, and human presence, and a state-decoding task that discriminates perception from recall. Within-subject validation results show numerically similar performance for several model--task combinations, alongside higher perception accuracy in two coarse-grained comparisons after multiple-comparison correction. In contrast, perception and recall are highly separable at the state level under the present validation protocol. State accuracy remains high after alpha-band ablation, although this does not exclude ocular or other task-related contributions.