Learning Compositional Spatio-Temporal Video Grounding with Synthetic Curriculum
2026-08-31 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors highlight that current video grounding models struggle when given complex questions that require understanding multiple details and relationships in a video. They introduce a new task, CompSTVG, to test models on these harder queries. To help with this, they create a synthetic data engine that produces video scenes and questions with controlled difficulty, forming a benchmark called STVG-CompBench. Testing existing models showed they perform poorly on these complex tasks. The authors also propose a new training method, CurrSTVG, which helps models learn better by gradually increasing question difficulty.
Spatio-Temporal Video GroundingMultimodal Large Language ModelsCompositional QueriesScene GraphSynthetic Data GenerationBenchmark DatasetCurriculum LearningReinforcement LearningTemporal ComplexitySpatial Interference
Authors
Xingjian Wang, Shijian Wang, Yibo Wang, Zihao Yu, Runhao Fu, Xuelian Cheng, Zongyuan Ge
Abstract
Despite the impressive progress of recent MLLMs on spatio-temporal video grounding (STVG), existing evaluations and training data focus primarily on simple queries. They largely overlook the compositional queries prevalent in real-world scenarios, where a target must be disambiguated by jointly reasoning about its attributes and relations to other entities. To bridge this gap, we propose Compositional Spatio-Temporal Video Grounding (CompSTVG), a task that requires models to process complex textual queries where every intertwined attribute and relational cue is essential for disambiguation. To facilitate this task at scale, we build a synthetic data engine that leverages a spatio-temporal scene graph as a difficulty measure and casts difficulty-controlled query synthesis as a constraint programming problem, producing difficulty-graded data for both evaluation and training. Built on this engine, we introduce STVG-CompBench, a benchmark stratified by explicit difficulty levels that jointly capture temporal complexity and spatial interference. Evaluating 11 representative STVG models on STVG-CompBench reveals that current models perform poorly on compositional queries, exhibiting a sharp performance drop that is typically obscured by overall dataset-level averages. We further construct synthetic training data and propose CurrSTVG, a curriculum reinforcement learning framework that delivers consistent gains, with the largest improvements observed on the most challenging compositional queries.