Temporal Self-Distillation: Learning Visual State Tracking in Videos Without Supervision
2026-09-03 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors present S³T, a new method for tracking changes over time in videos without needing any labeled data. Their idea is to use a detailed view of a video clip as a 'teacher' and teach a simpler version of the model to predict what happens next based on fewer frames. This system trains itself without extra help or extra running time when it is used. Tests show that their approach improves accuracy on synthetic and real video datasets compared to previous methods. The improvements come from learning using only unlabeled video clips.
self-supervised learningvideo state trackingself-distillationtemporal samplingnext-token predictionteacher-student modelsynthetic datavision encoderzero-shot transferunlabeled data
Authors
Shravan Venkatraman, Wenshuai Zhao, Mohammad Hassan Vali, Arno Solin
Abstract
We introduce S$^3$T (Self-Supervised Self-Distillation over Time), which, to the best of our knowledge, is the first fully self-contained framework for continuous video state tracking. Our method treats temporal sampling density as privileged information, based on the hypothesis that a denser view of the same clip recovers the running state more accurately. This view serves as the teacher, while a sparse-view student with the same weights learns to match its next-token distribution. The model generates its own target, so training requires no labels, separate teacher, or reward signal, and adds no inference cost. On LLaVA-OneVision-2-8B, S$^3$T improves VSTAT accuracy by $+1.74$ as a single model, $+2.38$ with souping, and $+2.70$ with additional vision-encoder adaptation, while prior self-evolving methods leave state tracking largely unchanged. The capability learned from unlabeled synthetic clips transfers to real videos, improving performance by $+7.95$ on VSTAT-YouTube state-tracking questions and $+4.50$ on MVBench Action Count.