Lightweight system locates actions in video using simple models
Pocket-STVG: lightweight architecture for Spatio-Temporal Video Grounding
Computer Vision and Pattern RecognitionArtificial IntelligenceMultimedia
Summary
Spatio-Temporal Video Grounding means finding where and when something happens in a video based on a sentence describing it. The authors created Pocket-STVG, a small and fast system that combines existing efficient components to do this job without needing huge or complicated models. It works well even with less training, making it easier to find video segments quickly. Their approach also creates video data that can be stored and accessed without reprocessing for every new search.
What this means in practice
- •For video platform engineers: Implement fast video search tools that locate actions or events based on natural language queries with low computational resources.
- •For mobile app developers: Build video apps that identify and highlight scenes matching spoken or typed descriptions without overloading phone memory or CPU.
Authors
Alberto Presta, Michal Byra, Grzegorz Stefański, Karol Szurkowski, Eryk Kołodziejczyk, Krzysztof Arendt
Abstract
Spatio-Temporal Video Grounding (STVG) aims to localize the spatio-temporal tube in a video corresponding to a natural language query. While recent methods achieve strong performance in fully supervised, weakly supervised, and zero-shot settings, they typically rely on computationally expensive architectures, complex training pipelines, or multimodal large language models. We present Pocket-STVG (P-STVG), a lightweight cascade architecture that addresses STVG by combining efficient pre-trained components instead of large end-to-end models. P-STVG integrates a temporal-aware video encoder based on MobileViCLIP, a spatial encoder-decoder derived from MDETR, and a shared aligned text encoder. Temporal localization is performed through either a lightweight 1D U-Net or a simple thresholding strategy, enabling the same framework to operate in both weakly supervised and zero-shot settings. Furthermore, video representations are precomputed independently of the query, yielding an indexing-friendly pipeline for efficient inference and large-scale video collections. Despite requiring fewer than 90M parameters, P-STVG performs on par with weakly supervised methods and improves on earlier zero-shot approaches at a fraction of their memory and computational cost, establishing a favorable performance-efficiency trade-off for STVG.