Frozen foundation models improve dense optical flow estimation accuracy

Foundation Model Driven Optical Flow

Computer Vision and Pattern RecognitionRobotics

Summary

Optical flow helps track how things move between video frames, which is useful in many technologies like self-driving cars and video editing. This paper shows a way to improve optical flow by using features from existing, powerful image and depth models without changing their inner workings. The authors combined these reused features with a new matching system and trained it to predict motion accurately. Their approach performs well compared to earlier methods, demonstrating that frozen foundation models can effectively support motion estimation with limited extra training.

What this means in practice

Authors

Praroop Chanda, Suryansh Kumar

Abstract

We investigate how frozen pretrained visual and depth representations support dense optical flow when coupled with a supervised matching network. Building on TTSFlow, this paper studies Dense Prediction Transformer (DPT) feature decoding and one coarse-to-fine refinement stage. Our method fuses features from frozen DINOv2 and Depth Anything V2 models and estimates optical flow through a trainable transformer-based matching pipeline. After optical-flow training on FlyingChairs and FlyingThings3D, it achieves 2.37 endpoint error (EPE) on the Sintel training split, Final pass, compared with 2.52 for the FlowSeek (T) configuration reported here. Our RoCo-Spring 2026 entry, RoCo-29.VSAI, obtains a robustness-balanced score of 1.1038, Spring EPE of 1.411, and RobustSpring proxy EPE of 4.739, with a lower balanced score than seven of the twelve listed reference baselines. These results characterize the complete pipeline, which reuses frozen foundation backbones and performs one refinement stage. Code is available at: \url{https://github.com/Visual-and-Spatial-AI-Lab/FFlow}
View PDFOpen arXiv