VideoMSN improves video learning using image transformers
Image Classifiers are Efficient Self-Supervised Video Representation Learners
Computer Vision and Pattern RecognitionMachine Learning
Summary
Learning useful information from videos without labels is usually slow and requires complex models. The authors created VideoMSN, which treats videos like special images made from several frames in a grid and uses existing image models to learn from them efficiently. VideoMSN cleverly hides parts of the frames or entire frames to learn both how things look and move, without needing to reconstruct the video. This method trains much faster than previous ways and works well even when only a few labels are available.
What this means in practice
- •For video analytics teams: Create efficient video feature extractors using image transformers for faster training with less labeled data.
- •For mobile app developers: Develop lightweight video understanding features that can be pre-trained quickly and perform well in scenarios with few labeled videos.
Authors
Owais Iqbal, Sudipta Sarkar, Shyam Marjit, Omprakash Chakraborty, Anirban Chakraborty, Abir Das
Abstract
We introduce VideoMSN, a Masked Siamese Network framework for efficient self-supervised spatio-temporal representation learning in videos. Instead of relying on heavy 3D architectures or reconstruction-based autoencoders for learning with unlabeled data, we repurpose standard image Vision Transformers by representing videos as super images which are grids composed of frames sampled from videos. From each super image, we construct two views: one with spatial patch masking and the other with temporal frame masking, ensuring no information leakage across frames. A shared Vision Transformer (ViT) encoder aligns their embeddings using a masked Siamese loss, capturing both motion and appearance cues without reconstruction. Our decoder-free formulation leverages an image foundation model towards efficient video representation learning. Starting from pretrained DINO-v3 and DeiT-v3 image encoders, VideoMSN achieves state-of-the-art performance on Kinetics-400, UCF101, and HMDB51 while requiring up to $32\times$ fewer and $160\times$ fewer video pretraining epochs compared to prior video self-supervised learning methods. Our proposed approach also shows strong performance in low-shot classification, confirming the transferability of the learned representations in a label-scarce scenario. Project Page: https://cvir.github.io/projects/videomsn.