Behavior pack optimization improves video question answering accuracy significantly
Behavior Pack Optimization for Video MLLM Post-Training
Computer Vision and Pattern Recognition
Summary
Video language models often get video questions right by guessing from visuals and language hints, not by understanding the video over time. The authors found this happens because existing training rewards only focus on single answers, ignoring how answers should change with altered videos. They created a method called Behavior Pack Optimization that trains the model to give consistent answers across different versions of a video clip, improving understanding of time-related details. Their approach led to better accuracy on video question tests and improved the model's ability to say when it doesn't have enough evidence to answer.
What this means in practice
- •For video software developers: Improve video question answering systems by training models to respond consistently across different video variants, boosting temporal understanding and reliability of answers.
- •For multimodal ai engineers: Enhance multimodal large language models with stable post-training techniques that increase sensitivity to key video evidence and reduce incorrect confident answers.
Authors
Zhaolu Kang, Shiyu Liu, Tailong Luo, Wei Zhang, Yingjie He, Lei Wei, Guansu Wang, Liang He, Siheng Wang, Guangyuan Dong, Jiaqi Su, Shuang Chen, Haoyu Ji, Qishi Zhan, Kaiyue Zhou
Abstract
Video multimodal large language models (MLLMs) keep climbing video question answering benchmarks, yet shuffling the frames, masking the segment that supports the answer, or occluding the target object barely changes their predictions. The accuracy rests on appearance and language priors, not on the temporal evidence the question asks for. We trace this to the unit of post-training: rewards are computed on a single response to the original clip, so the model is never asked to behave consistently across views. We propose Behavior Pack Optimization (BPO), which replaces the single response with a behavior pack of outputs across counterfactual views chosen by question type, scored jointly. The pack reward asks for stability when the intervention is irrelevant, sensitivity when key evidence is removed, and abstention when no evidence remains. To keep this objective stable at small pack sizes, BPO uses an anchor-relative advantage: the response on the original view serves as a per-prompt reference instead of a group mean over mixed views. On TempCompass, MVBench, and NExT-QA, BPO improves the macro accuracy of Qwen2.5-VL-7B-Instruct by 4.7 pp, the temporal-hard subset by 7.8 pp, and abstention F1 by 20.0 pp over a budget-matched vanilla GRPO baseline from the same SFT checkpoint. The gains transfer to Video-MME, LongVideoBench, and to LLaVA-Video-7B; ablations confirm they follow the view sets, not the rollout count. We hope this pack-level perspective offers a useful starting point for the video MLLM and multimodal post-training community as the field moves toward evidence-grounded video reasoning.