Video diffusion adaptation improves motion tracking without hurting quality
AcrossVAM2.0: Conflict-Aware Point Supervision for Parameter-Efficient Video Diffusion Adaptation
Artificial Intelligence
Summary
Adapting video generators to track moving surfaces better can sometimes harm the video quality learned by large models. The authors study this problem in a large text-to-video model and propose a method called AcrossVAM2.0 that carefully balances learning motion tracking points while preserving the original video quality. Their experiments show that this method improves tracking accuracy without significantly lowering overall video quality, but improvements are not consistent for every measure. They share their full approach and outputs for others to study.
What this means in practice
- •For video generation engineers: Improve how video generation models track object motion by adding conflict-aware adaptation for more consistent surface movement without major quality loss.
- •For computer vision developers: Enhance video analysis tools that rely on motion flow data by using improved point correspondence from adapted video diffusion models.
Authors
Nan Wu, Yee Hung Hong
Abstract
Explicit point correspondence is attractive for adapting video generators: it describes how visible surfaces move, but can overwrite the appearance prior learned by a large pretrained model. We study this tension in a 5.01B-parameter text-to-video diffusion transformer. ACROSSVAM2.0 combines a confidence-weighted track-and-visibility head, zero-initialized residual feedback, and asymmetric PCGrad that removes only the auxiliary component opposing the primary flow-matching gradient on shared LoRA parameters. The design preserves the generator exactly at initialization and protects its primary update during adaptation. In a four-way ablation trained for 1,000 steps with three independent seeds, the full method lowers KLT forward-backward cycle error in all three seeds relative to a matched flow-only continuation (mean seed-level difference -0.00711, SD 0.00505), while robust sharpness remains positive in all three. An expanded 24-prompt evaluation (216 videos) yields 9.7% lower aggregate CoTracker3 cycle error than the matched control and near-matched performance on six VBench dimensions; however, the seed-level tracking effect is heterogeneous and I3D-FVD regresses from 0.1824 to 0.2022. These results support a bounded claim: conflict-aware adaptation improves correspondence consistency without broad perceptual degradation, but does not uniformly improve video quality. We release equations, pseudocode, matched outputs, source tables, and artifact provenance.