Self supervised learning improves for continuous video streams
I Have a Stream: Making Self-Supervised Learning Work on Continuous Video
Computer Vision and Pattern Recognition
Summary
Training AI to understand video usually involves mixing up images from many videos and showing them repeatedly. The authors studied training AI on long, continuous video streams in the order they occur, more like how babies learn. They found that many existing learning methods struggle, mainly because the video frames in each training batch look very similar. They propose a method called StreamMAE that changes how the video is fed into the AI and selects video parts with movement, resulting in better learning on continuous videos.
What this means in practice
- •For video analysis engineers: Train AI models more effectively on long video streams without needing shuffled or repeated data.
- •For autonomous vehicle teams: Improve visual perception training by using raw continuous video from urban environments.
Authors
Ivan Martinović, Lukas Knobel, Yuki M. Asano
Abstract
Self-supervised learning draws inspiration from infant visual development, yet standard training pipelines bear little resemblance to it: images are independently sampled and globally shuffled across epochs. We study self-supervised learning from continuous video streams, where frames are consumed in temporal order using strict sliding-window batches, without global reshuffling or multi-epoch replay. To this end, we construct WT++, a 95-hour urban walking-tour video dataset for streaming pretraining. Combined with a comprehensive evaluation suite we find that contrastive and distillation-based methods struggle in this setting, while MAE is more robust but still falls short of standard i.i.d. pretraining. We find that high inter-batch similarity, caused by sliding-window consumption across consecutive batches, does not explain this gap. The main challenge is high intra-batch similarity, where frames within each batch are near-duplicates. To mitigate this, we propose StreamMAE, which preserves the core MAE reconstruction objective while adapting the input pipeline with stream-aware regularization and motion-biased crop selection. StreamMAE outperforms streaming baselines, matches i.i.d. MAE trained on the same video data, remains competitive with ImageNet-pretrained MAE, and scales positively as the pretraining stream grows from 12 to 95 hours.