Large video models struggle with visual questions over text options
Not Another Text Benchmark: Putting the "Visual" Back in Visual Question Answering for Large Video Models
Computer Vision and Pattern Recognition
Summary
Many advanced video models are good at answering questions about videos when the answers are given as text choices. But the authors looked at how well these models do when the questions and answers are visual instead of text-based. They created three new tests focusing on understanding videos visually, such as finding specific frames, predicting future video scenes, and spotting changes in memory. Their tests show that current models have a hard time reasoning visually, highlighting a gap in video understanding that needs improvement.
What this means in practice
- •For video platform developers: Improve automated video content tagging by evaluating model visual reasoning with new benchmarks focusing on frame retrieval and event prediction.
- •For computer vision engineers: Enhance model training by identifying weaknesses in visual question answering tasks through targeted benchmarks that require reasoning over visuals, not text.
Authors
Rwiddhi Chakraborty, Yinong, Wang, Cheng Zhang, Fan Bai, Zhuoran You, Michael Kampffmeyer, Yong Jae Lee, Fernando De la Torre, Robert Jenssen
Abstract
Large video models have exhibited impressive performance on a wide range of visual question answering tasks, owing to the rise of powerful, pretrained text and vision encoders. The usefulness of such models have also been demonstrated on a wide range of benchmarks, with an important caveat - the dominant approach in these benchmarks evaluates multiple choice reasoning via text options. This is a natural way to test text-based reasoning in these models, and has led to significant insights regarding model behavior in the community. In this work, we ask a different question - what happens when the evaluation modality is visual, rather than text? We introduce three new vision-centric evaluation benchmarks in temporal frame retrieval, video future prediction, and causal memory distortion, all designed around evaluating visual understanding capabilities in large video models. Our approach complements the existing approaches to evaluate video understanding in frontier models. We show that current frontier models exhibit significant weakness when attempting to reason through visual queries, rather than text. We conclude with an extended analysis section that provides pointers for future improvements in visual understanding for large video models.