Video representations improved by learning object layouts in scenes
Video-STLayout Pre-training
Computer Vision and Pattern Recognition
Summary
Understanding actions in videos is hard because scenes are complex and involve many objects moving together. The authors found a way to teach computers to watch videos while also paying attention to where objects are and how they move over time. They do this by detecting objects first and then training the video understanding system to align with these object layouts. This approach helps computers recognize activities more accurately, especially in complicated scenes with many interacting parts.
What this means in practice
- •For video analytics engineers: Improve activity recognition models by integrating object layout information during training, leading to better understanding of complex video scenes.
- •For robotics software teams: Enhance robots’ perception by training video models to use object layouts, aiding robots in interpreting dynamic environments more accurately.
Authors
Akash Abdu Jyothi, Greg Mori
Abstract
In recent years, pre-training has become fundamental to learning effective video representations, enabling strong transfer to downstream tasks. A popular framework in pre-training involves aligning features of a video encoder with that of another modality, for example, language or audio. We introduce Video-STLayout pre-training, a novel strategy for obtaining rich video representations informed by spatio-temporal layout of object bounding boxes. Object layouts can easily be obtained by applying an off-the-shelf object detector on the video frames. Our method uses a contrastive loss to align video features with the layout features from a trained layout encoder. We show the effectiveness of our approach in the task of activity recognition in complex scenes.