Optimal transport improves frame selection for long video understanding
A Principled Optimal Transport Framework for Frame Selection in Long Video Understanding
Computer Vision and Pattern Recognition
Summary
Long videos can be hard for AI models to understand because the models can only look at a limited number of frames. The authors looked at how to pick the most important frames for understanding these videos. They created a new method that uses a math idea called optimal transport to fairly assign importance from different clues to the frames. This method works better than older ways because it carefully balances keeping key information and ignoring redundancy. Their tests on video tasks show their approach picks frames that help the AI understand videos more accurately.
What this means in practice
- •For video analysis teams: Improve automated selection of key frames to better understand long videos with limited AI model context.
- •For security monitoring operators: Select relevant video frames efficiently from long surveillance footage for faster incident review.
Authors
Ziwei Zhao, Yuhang Liu
Abstract
Long-video understanding with multimodal LLMs is fundamentally constrained by the mismatch between video length and the model's limited visual context budget, making frame selection essential for practical reasoning. Existing query-aware methods typically rely on heuristic pipelines: they assign one-dimensional query-conditioned scores to frames and then apply sampling, ranking, or extraction rules to form the final subset. While effective in practice, such methods are generally not derived from an explicit optimization objective and struggle to capture the heterogeneous evidence required for complex reasoning. In this paper, we revisit long-video frame selection from a principled optimization perspective. We formulate it as a structured evidence-allocation problem and propose OTFS, an optimal transport framework that maps multiple evidence sources onto video frames. To reflect the asymmetry of this problem---important evidence should be preserved, whereas redundant frames need not absorb mass---we instantiate this view with a semi-unbalanced entropic optimal transport objective and an efficient solver, OTFS-Sinkhorn. Combined with an exact dynamic program for coverage-aware subset extraction, OTFS yields a practical two-stage training-free pipeline. We further show that, in the single-source case, our formulation induces a Gibbs-type distribution over frames, making Q-Frame's temperature-scaled distribution a special case of our framework while also providing a broader perspective for understanding related prior methods. Experiments on three long-video understanding benchmarks show that OTFS consistently outperforms strong training-free baselines. Codes are available at https://github.com/barry664/OTFS.