Multimodal models improve long video answers by fixing bias errors
Diagnosing and Correcting Bias in MLLM for Long Video Understanding
Computer Vision and Pattern Recognition
Summary
Answering questions about long videos is hard because important moments are scattered, and models can miss key parts due to limited memory. The authors show that models often rely on misleading language clues or obvious but irrelevant pictures instead of the right video details. They create a new test called S-MME to find these shortcut mistakes. Then, they offer a method called CLEAR that checks what parts of the video and question cause bias and adjusts the model’s reasoning at test time to reduce these errors. Their method works across different models and video tests, making answers more accurate and reliable.
What this means in practice
- •For video platform developers: Enhance video question answering features by reducing model errors caused by misleading cues in long videos.
- •For multimedia ai engineers: Integrate a bias correction framework to improve robustness of large models processing sparse long video data.
Authors
Xusheng Liang, Jianqiao Sun, Hao Zhang, Yulei Niu, Hengshuang Zhao, Jiawei Ma
Abstract
Long-video question answering is challenging since the answer often hinges on a few decisive moments scattered throughout the video, while memory constraints force Multimodal Large Language Models (MLLMs) to sample frames at extremely low rates, resulting in sparsity that risks omitting key evidence. Though aggressively enlarging the context size is intuitive, we argue that performance remains fundamentally constrained by the bias, \emph{i.e.}, the misleading cues caused by spurious \textit{linguistic} correlations and salient yet irrelevant \textit{visual} observations. In this paper, we first introduce {S-MME}, a benchmark designed to systematically diagnose shortcut-biased behavior in MLLMs. To mitigate this failure, we further propose a training-free framework, Counterfactual Long-video Evidence-Aware Reasoning (CLEAR). By treating the full video-question pair as a complete view, CLEAR estimates bias effects via augmented views by only keeping the question or locally salient visual clips. Then, we intervene hidden states to mitigate the discrepancy at inference such that the correct answer can be predicted confidently. With experimental studies across five benchmarks, we show that our method is generalizable and can be integrated in different MLLMs, yielding consistent gains in prediction and robustness. We hope CLEAR can offer a complementary direction in test-time scaling for long-video understanding.