Tempo-Zero improves video reasoning using self-play and executable environments
Tempo-Zero: Recursive Self-Play in Executable Worlds for Video Reasoning
Computer Vision and Pattern Recognition
Summary
Understanding how events happen over time in videos is hard because labeled examples are costly to make. To address this, the authors propose Tempo-Zero, a method where a single AI learns by creating and answering its own questions in a simulated world that produces perfect answers. This approach lets the AI improve itself without needing real labeled videos, and it performs better than prior methods on many video understanding tasks, especially those that require knowing when events happen. Tempo-Zero helps AI learn to reason about videos by practicing in a controllable and well-labeled environment.
What this means in practice
- •For video analytics developers: Develop systems that better understand timing of events in videos without relying on costly annotated data.
- •For simulation platform builders: Create video datasets automatically by evolving rules in simulated environments for training video reasoning models.
Authors
Jingmin Zhu, Yanyan Zou, Xiuyu Li, James Bailey, Dinh Phung, Tao Yu, Yunyun Hou, Jun Liu, Jun Zhou, Sirui Wang, Nan Duan, Jiachen Liu, Qiuhong Ke
Abstract
Video reasoning requires understanding how events unfold over time, yet videos with accurate temporal annotations are expensive to collect. Self-play offers an alternative by letting a model generate and solve its own tasks without manual annotation. However, existing video self-play methods train on fixed sets of recorded videos, so they cannot alter the underlying event dynamics and rely on noisy pseudo-labels derived from the model's own outputs, such as majority voting, that risk reinforcing shared model errors. To overcome these limitations, we introduce Tempo-Zero, a self-improvement framework that closes a loop of executable environments, execution-verified self-play, and environment evolution, where executing a program renders a video with the ground-truth option and interval of every question. A single policy plays both Questioner and Solver, proposing questions and answering them with timestamped evidence lines, and is trained on execution-grounded rewards instead of model consensus. The Solver's performance steers a curriculum over tasks and difficulty levels, and the policy selects rule changes that enrich the event dynamics of later rounds. Despite being trained solely on simple programs, without supervised fine-tuning or external video data which prior methods typically require, Tempo-Zero outperforms the base models and recent self-play baselines on average across twelve real-world video benchmarks at different model scales, with particularly strong gains on temporal grounding tasks. Project page: https://alchemist-jo.github.io/tempo-zero/