Vision language benchmark compression saves compute while keeping model rankings
Hierarchical Compression of Vision-Language Model Benchmarks
Machine LearningArtificial IntelligenceComputation and LanguageComputer Vision and Pattern Recognition
Summary
Evaluating vision-language AI models is becoming very costly because benchmarks are growing larger and new models keep coming out. The authors developed PRIMEBench, a smart way to shrink these tests dramatically without changing how models rank against each other. Their method removes easy or redundant items, chooses representative tests per category, and prunes samples by measuring how much models differ and how much images matter. This approach cuts more than 97% of test items while preserving evaluation quality, letting users save lots of compute time. Their work also helps understand how to design better benchmarks as more models appear.
What this means in practice
- •For ai development teams: Evaluate new vision-language models with minimal computation by using highly compressed yet representative benchmarking tests.
- •For cloud infrastructure planners: Plan resource allocation more efficiently by cutting down the cost and time needed for vision-language model evaluations.
Authors
Hyunjong Ok, Seunggu Kang, Jaeho Lee
Abstract
Thorough evaluation of vision-language models (VLMs) has become prohibitively expensive, as benchmarks span an ever-broader spectrum of capabilities and new models arrive at a relentless pace. Benchmark compression methods that preserve model rankings at a fraction of the cost are well studied for language models, but for VLMs the question remains under-explored. We present PRIMEBench (Pruning Redundant Items for Multimodal Evaluation), a vision-aware hierarchical benchmark compression framework that substantially reduces evaluation cost while preserving model rankings. This hierarchical framework operates in four stages: data cleaning to remove items answerable without the image and all-correct items, category representative selection to pick one benchmark per capability category, item pruning with Vision-Aware Variance (VAW), and category-count pruning. VAW combines inter-model variance with a vision-dependence score computed from multimodal embeddings alone, while encouraging coverage of diverse items within each benchmark. On models held out from item selection, it has the highest mean fidelity at the released 5% retention. The hierarchical design lets practitioners stop at any stage to match their compute budget; the released suite removes over 97% of items while preserving model rankings. Beyond compression, our analyses show how VLM evaluation behaves as model panels grow and evolve, providing guidance for designing future benchmarks that are more efficient, robust to model turnover, and explicit about the limits of evaluation-side pruning.