Compact vision language models get better benchmark with seven axis scores
PRISM-VLM: A Multi-Axis Discriminative Benchmark for Compact Vision-Language Models
Computation and Language
Summary
Evaluating small vision-language models usually relies on a single overall accuracy score, which can hide important differences between models. The authors created PRISM-VLM, a new benchmark that tests models across seven different aspects related to common failures and capabilities. This multi-axis approach gives a more detailed picture of how models perform differently, even when their overall scores look similar. It can highlight specific weaknesses like tendency to agree with prompts regardless of truth, which standard benchmarks miss.
What this means in practice
- •For machine learning engineers: Identify specific behavioral strengths and weaknesses of compact vision-language models to select the best model for different multimodal tasks.
- •For ai product developers: Improve product performance by understanding subtle failure modes in vision-language models that standard accuracy metrics overlook.
Authors
Sanghee Park, Kee-Eung Kim
Abstract
Compact vision-language models (VLMs) now power a growing share of multimodal applications. The benchmarks used to compare them, however, inherit a frontier-centric design: each model is reduced to a single accuracy number, narrowing the inter-model gap on saturated suites and pressing models into low-score bands on harder ones. We introduce PRISM-VLM, a multi-axis discriminative benchmark that scores every item along seven axes covering the recurring failure modes (task quality, behavioral robustness, and capability bottlenecks) and combines them into a single PScore, with items recycled from fifteen public benchmarks. Across compact VLMs from the past two years, PScore separates model pairs more reliably than prior single-axis benchmarks under an item-level paired bootstrap, and surfaces behavioral differences these benchmarks average away. Even models with statistically indistinguishable PScores diverge sharply along the per-axis profile, particularly on sycophancy, which is nearly orthogonal to single-prompt accuracy. We will release the full pipeline, prompts, and per-item annotations.