Video diffusion transformers improve object matching across views and time
Repurposing Video Diffusion Transformers for Cross-View Temporal Object Correspondence
Computer Vision and Pattern Recognition
Summary
Tracking the same object in videos from different cameras and moments is tricky because objects can look different or be hidden. The authors propose a system called XTrack that uses a special type of video model to keep track of objects both across camera views and over time. They improved the model to avoid mixing up similar objects and created a way to expand limited labeled data for better training. This approach helps keep identities consistent even when objects change appearance or get partially hidden.
What this means in practice
- •For video analytics developers: Track the same object reliably across multiple camera angles and time in surveillance or sports footage.
- •For augmented reality engineers: Maintain consistent object identities when blending views from different cameras in AR experiences.
Authors
Siyoon Jin, Dahyun Chung, Honggyu An, Sangbeom Lim, Wonjin Nam, Jaewoo Jung, WonJun Moon, Seungryong Kim
Abstract
Cross-view object correspondence identifies the same object across views. However, existing methods often rely on frame-wise matching or sequential tracking, which can propagate correspondence errors under appearance changes or occlusions. Therefore, we introduce cross-view temporal object correspondence, which requires preserving object identity across both views and time from a query mask. To address this, we propose XTrack, a unified framework which repurposes a pretrained video DiT with semantic and spatiotemporal priors for RGB-to-mask latent transport, jointly modeling cross-view matching and temporal tracking. However, naive video DiTs do not guarantee accurate cross-view grounding and often attend to semantically similar yet incorrect objects across views. Thus, we introduce focal cross-view attention alignment, hard negative conditioning, and local context crop strategies to ensure reliable grounding. Finally, beyond these architectural enhancements, we also tackle the lack of dense annotation by introducing a pseudo-label curation pipeline that converts sparse Ego-Exo4D annotations into dense supervision to enable effective video-level training. Our results demonstrate the effectiveness of XTrack as a unified framework, ensuring robust temporal consistency and identity preservation.