Watching scenes change improves image and video recognition features
W2Rep: Learning Visual Representations by Watching the World Change
Computer Vision and Pattern Recognition
Summary
Images show a single moment, but videos show how things change over time. The authors developed W2Rep, a method that learns to predict how an image changes using both one picture and its surrounding video context. This helps build features that work well for recognizing objects in both single images and videos. Their experiments show that using changes over time helps create better visual understanding without losing the ability to analyze still images.
What this means in practice
- •For computer vision engineers: Improve image and video recognition systems by training feature extractors that capture changes over time.
- •For video content analysis teams: Enhance analysis tools that require understanding both single frames and temporal changes in video data.
Authors
Wen Huang, Hang Guo, Jiarui Yang, Zheng Liu, Tao Dai, Shu-tao Xia
Abstract
Images capture the world at one moment, whereas video reveals how it changes. Image self-supervision learns spatial structure from a single moment, while video methods commonly learn temporal relationships inside a representation computed jointly from several frames. We ask whether watching a scene change can instead improve features available from one image without sacrificing the ability to represent video. We introduce W2Rep, a masked feature-prediction framework in which an independently encoded source image participates in prediction at the same or another moment. The predictor is conditioned on visible video context, the queried location, and the signed time interval between source and target. This gives the cross-frame objective two complementary roles: the image path learns features that remain useful across time, while the video path must gather evidence that is missing from the source image. Across model scales and downstream tasks, W2Rep improves frozen and fine-tuned recognition under our comparison protocol, while joint video encoding provides further gains over frame-wise aggregation. Controlled experiments show that these gains depend on directly updating the source-image features and on using both video context and temporal displacement. Overall, change across a video can supervise a visual encoder whose representations remain useful at either image or video granularity. Code is available at~\href{https://wenooi.github.io/W2Rep}{https://wenooi.github.io/W2Rep}.