Brain-Aligned Multi-Stream Video Transformers with Sparse Self-Selection
2026-07-20 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionMachine Learning
AI summaryⓘ
The authors designed a new type of video transformer inspired by how primate brains process visuals, using a sparse selection method instead of dense self-attention to make the model more efficient. They created two separate pathways: one focusing on detailed images at fewer frames and another capturing movement at more frames, combining these before making predictions. Their model performs well on standard video tasks, is faster, and better at dealing with image distortions compared to similar models. Additionally, the model's internal activity closely matches brain signals recorded from humans watching the same videos, indicating it processes visual information in a brain-like way.
video transformerself-attentionsparse winner-takes-allprimate visionsplit-and-fuse architecturehigh-resolution streamlow-resolution streamrepresentational similarity analysisEEGKinetics-400 dataset
Authors
Amir Hosein Fadaei, Mahyar Maleki, Mohammad-Reza A. Dehaqani
Abstract
Modern video transformers typically ignore principles from primate vision and are rarely evaluated against neural data, limiting their biological interpretability. We introduce a sparse winner-takes-all token selection module that replaces dense self-attention to improve efficiency and approximate competitive routing observed in biological visual circuits. We further propose a neuro-inspired split-and-fuse video transformer which uses two complementary pathways: a high-resolution, low-frame-rate "what" stream and a low-resolution, high-frame-rate "where" stream, fused before classification. On Kinetics-400 and Something-Something V2, our best variant operates on the Pareto frontier of accuracy versus inference time among models of comparable scale and pretraining, and showing improved robustness to spatial perturbations. Using representational similarity analysis between model embeddings and time-resolved EEG recordings for the same video stimuli, our model attains a peak brain-model correlation of 0.18 (about 78% of the noise ceiling) and consistently outperforms strong video transformer baselines, suggesting that pathway specialization and sparse competition are useful inductive biases for efficient, brain-aligned video understanding.