Diffusion trajectory modeling improves matching of image parts over time
Diffusion Trajectory Modeling for Semantic Correspondence
Computer Vision and Pattern Recognition
Summary
Matching parts of images that show the same things can be hard, especially when pictures look different or have changes. This paper finds that during the process where a special kind of AI creates images step by step, each small part in the image changes in a meaningful way over time. The authors created a method called diffusion trajectory modeling that looks at how each part changes through these steps to find better matches between images. Their tests show that watching these changes over time helps computers understand which parts of different images correspond to each other more accurately.
What this means in practice
- •For computer vision engineers: Improve algorithms for matching corresponding regions across images using temporal features from diffusion models.
- •For augmented reality developers: Enhance object tracking by exploiting temporally evolving patch features during scene generation for better spatial correspondence.
Authors
Yusung Choi
Abstract
Diffusion models generate images through an iterative diffusion process, and recent studies have demonstrated that the intermediate feature maps produced during this process contain rich visual representations, leading to their adoption across a variety of downstream tasks. However, most existing approaches are limited to either using a single feature map at a specific timestep or aggregating feature maps across multiple timesteps. We observe that intermediate representations in the diffusion process form meaningful trajectories along the time axis. In particular, the representation of each spatial patch evolves progressively throughout the generative process, encoding semantics that are difficult to capture from static snapshots alone. This observation motivates the need to treat diffusion representations as temporally structured trajectories rather than static snapshots. To this end, we propose Diffusion Trajectory Modeling (DTM), a framework that interprets the temporal evolution of each spatial patch as a trajectory and leverages it for semantic correspondence. By effectively modeling patch-wise trajectories generated across multiple timesteps, DTM captures correspondence cues that prior methods are not designed to capture. We further demonstrate empirically that spatially corresponding patches form similar trajectory patterns throughout the diffusion process, suggesting that the temporal axis of diffusion carries semantic information. Experiments on SPair-71k, SPair-U and AP-10K show that DTM achieves strong performance, presenting a new perspective for exploiting diffusion representations from a trajectory-centric viewpoint.