VCIFBench: Evaluating Complex Instruction Following for Video Understanding
2026-06-03 • Computation and Language
Computation and Language
AI summaryⓘ
The authors created VCIFBench, a new test that checks how well large language models understand videos when given complicated instructions with lots of rules to follow. They designed this test to see if models can meet strict demands about what to say and how to say it in their answers. The authors tested 10 different models and found that it's still hard for them to meet all these rules at once. They also showed that training models with VCIFBench data can help them follow instructions better.
multimodal large language modelsvideo understandinginstruction followingbenchmarkconstraint satisfactionDPO trainingpreference datasetverification pipeline
Authors
Huangchen Xu, Yuan Wu, Yi Chang
Abstract
Multimodal large language models have made rapid progress in video understanding, yet existing benchmarks largely rely on simple prompts and provide limited evidence about whether models can satisfy explicit output constraints. We introduce VCIFBench, a benchmark for evaluating complex instruction following in video understanding. VCIFBench constructs constraint-rich instructions from both benchmark-adapted and directly video-grounded prompts, covering content, format, style, and structure requirements, and evaluates model outputs with a hybrid verification pipeline. The benchmark contains 306 satisfiable test instructions, a 540-pair DPO preference dataset, and a 30-item conflict diagnostic subset. Experiments on 10 MLLMs show that joint constraint satisfaction remains challenging. We further show that DPO training on VCIFBench data can improve instruction-following performance.