Multimodal AI benchmarks can report different scores from same answers
Same Outputs, Different Scores: Auditing the Unchecked Graders of Multimodal LLM Benchmarks
Artificial IntelligenceComputation and LanguageComputer Vision and Pattern Recognition
Summary
Many tests for AI models that understand pictures and text score answers by converting what the model writes into right or wrong. The authors found these scoring methods often change the reported scores a lot even when the model always gives the same answer. Some scoring rules guess answers or ignore certain responses, which can hide how good a model really is. They suggest using multiple scoring methods and reporting how often a model's answer doesn’t fit the scoring rules to get a clearer picture.
What this means in practice
- •For ai system developers: Improve AI model testing by applying multiple scoring methods and reporting discrepancies to better understand model performance.
- •For machine learning benchmark designers: Design benchmarks that include measures of how often models’ outputs don’t match scoring rules and provide multiple scoring strategies for transparency.
Authors
Ahmad Jarjis Hasan, Jonathan Lwowski
Abstract
Every multimodal-LLM multiple-choice benchmark ends with a scoring rule that converts a model's response, often free-form text rather than a clean option letter, into a right/wrong verdict, buried in its code. Unlike the dataset, this rule is chosen without justification and left unexamined, yet it sets the published number. We reproduce the official scorers of eight peer-reviewed benchmarks on five open vision-language models (VLMs) that fit a single 16 GB GPU, and introduce a blinded human-plus-LLM-judge adjudication (Cohen's kappa=0.956) that recovers what the model actually answered. The scoring rule alone moves accuracy by up to 22 points on identical outputs, yet exactly 0 wherever a model emits clean option letters, so it hides on strong-model leaderboards. It is the fallback and not the rambling that does the damage: the benchmark with the highest non-compliance we measure has an artifact of exactly zero, because its scorer abstains instead of guessing; a reasoning model with non-compliance 0.000 still swings 9.5 points with which regex match of its trace the parser keeps. No standard rule is reliable: snapping a non-answer to its closest option credits answers never given yet buries correct ones, abstention discards answers stated in words, and likelihood scoring follows answer-token priors, so a single grade conflates capability with luck rather than measuring either. We release all code, labels, and the protocol, and argue benchmarks should report a non-compliance rate and at least two rules, not one unexamined grade.