Video models give unstable answers when frame timing shifts slightly
Stable Scores, Unstable Answers: Frame Phase and Option Order in Video Multiple-Choice Evaluation
Computer Vision and Pattern Recognition
Summary
When testing video-language AI models, small shifts in which video frames are used can cause the models to give very different answers. This happens even though the overall accuracy scores look similar. The researchers found that by combining results from multiple slightly offset sets of frames, they could reduce these answer changes. They also showed that how answer options are ordered can affect the model’s choices, so it’s important to report the frame timing and consider averaging to get more stable results.
What this means in practice
- •For video ai developers: Improve reliability of video model evaluations by using multiple frame offsets to reduce answer instability.
- •For benchmark designers: Design video-language benchmarks that report frame phase and use fusion methods to ensure stable accuracy comparisons.
Authors
Lichen Zhu, Yiheng Wang, Yueqian Lin, Hai "Helen" Li, Yiran Chen
Abstract
Video-language models are ranked by multiple-choice accuracy on frames from a uniform grid. The grid has two parameters, a rate and a phase, and benchmarks report only the rate. The phase moves answers: two deployed samplers differing only by a half-step phase offset answer 23.6% of questions differently while scoring within a point, and across four releases from two families shifting only the phase changes roughly one answer in five after controlling option order. PHASEFUSION decodes three offset grids and averages the option posteriors. The grids are the polyphase components of the dense grid. Fusion matches a 32-frame single pass in accuracy within a prespecified margin (logit-scored) and cuts the answers a half-step shift of all three grids changes from 18.2% to 10.1%. Option order, which changes only the presentation, is flagged instead by a one-pass answer margin. Report the phase convention with the budget, or marginalize it.