Video understanding improves with smart adaptive frame analysis

EVAThink: Evidence Sufficiency-Guided Training for Adaptive Thinking in Video Understanding

Computer Vision and Pattern Recognition

Summary

Understanding video content often depends on the key frames selected for analysis, but sometimes the frames don't provide enough clues for direct answers. The authors developed a method called EVAThink that teaches a video-language model when to rely on direct answers and when to use deeper reasoning based on how much useful information is present. This approach uses specially chosen video clips to train the model to adapt its thinking style, improving its performance on video question-answering tasks with limited frames. Results showed better accuracy even with fewer frames, suggesting smarter frame use helps computers understand videos more like people do.

What this means in practice

  • For video analytics engineers: Improve accuracy in video question-answering systems by enabling adaptive choice between direct answers and reasoning based on frame evidence quality.
  • For multimedia content platforms: Enhance content understanding with efficient frame usage to reduce computational cost while maintaining answer quality in video-based queries.

Authors

Changti Wu, Ruizhe Fan, Bin Wang, Lei Zhang, Yuxiang Zhang

Abstract

Under a fixed frame budget, video understanding with vision-language models (VLMs) depends on whether the sampled frames provide sufficient evidence for the question. Sufficient frame evidence supports direct answering, whereas less informative observations may benefit from reasoning that connects the available cues. We introduce EVAThink, which enhances video understanding through evidence sufficiency-guided training for adaptive thinking. First, Recoverable Evidence Gap Mining constructs evidence-dense and evidence-sparse video views that share the same question and answer under an identical frame budget. It retains pairs where sparse views yield lower direct-answer accuracy than dense views but benefit from explicit reasoning, providing supervision for response-mode selection. Second, Evidence-Guided Two-Stage Reinforcement Learning uses stage-specific rewards to guide training with evidence-based mode preferences and then refine response efficiency, while first-token advantage regularization mitigates mode collapse. Trained on only 6.7K curated samples, EVAThink consistently improves upon the base model across video understanding benchmarks under both 32- and 64-frame budgets, while outperforming strong baselines on average. The code, model, and datasets are available at \href{https://github.com/ChangtiWu/EVAThink}{EVAThink}.