Video language models learn better timing from teacher signals
Redirecting Teacher Supervision: Temporal-Response-Guided Latent Distillation for Video-Language Models
Computer Vision and Pattern Recognition
Summary
Training video-language AI models to understand timing in videos can be tricky. The authors found a way to improve this by guiding the student model using how it responds over time to small changes. This method, called Temporal-Response Guidance, adjusts the teacher’s feedback to better fit the student’s temporal perception. As a result, this approach improves the accuracy of a large video-language model on several video understanding tasks by nearly 3 percentage points.
What this means in practice
- •For video ai developers: Enhance video understanding models by improving how they learn from teacher feedback on timing-related features.
- •For natural language processing engineers: Improve joint video and language models’ temporal reasoning by incorporating refined internal supervision during training.
Authors
Xinkai Lu, Junli Zha, Haolan Wei, Jiahui Wang, Chao Zhou, Xiang Song
Abstract
On-policy distillation (OPD) transfers capabilities through teacher predictions on student-generated sequences, as studied by GKD and MiniLLM for language models and Video-OPD for video tasks. We ask how internal supervision can further support temporal learning. Direct hidden-state alignment does not explicitly use the student's temporal responses to choose a supervision direction. We propose Temporal-Response Guidance (TRG), which measures representation responses to local temporal perturbations, adds the teacher correction's projection onto their span back to the original correction, and restores its original norm. The projection is not normalized separately, and correction components outside the response space are retained. With an orthogonal tangent projector and a detached target, TRG changes the supervision direction while preserving the cosine-loss value and aligned-representation gradient norm at target construction. On TempCompass, Video-MME, MVBench, and Video-MMMU, TRG improves the 4B student's macro accuracy by 2.86 percentage points over the reported output-only OPD baseline.