HoloCosm improves compressed evaluation across multiple metrics
HoloCosm: Beyond Single-Metric Benchmark Compression Across Multiple Metrics
Computation and Language
Summary
Evaluating computer models can be expensive because it involves checking many test samples. The authors found that choosing a small set of samples guided by one performance measure often misses important info for other measures. They created HoloCosm, a method that picks sample sets that represent multiple evaluation metrics together, preserving model rankings better. This makes testing faster while keeping results reliable across different ways to judge models.
What this means in practice
- •For machine learning engineers: Reduce the cost of model testing by selecting evaluation samples that preserve performance across different metrics simultaneously.
- •For software qa teams: Speed up quality checks of AI components by using balanced sample sets that maintain consistent error and ranking measures across multiple evaluation criteria.
Authors
Yupei Lin, Zhongzhan Huang, Junxin Li, Guoming Ling, Shanshan Zhong, Hefeng Wu, Yukai Shi, Liang Lin
Abstract
Benchmark compression reduces evaluation cost by selecting a small set of samples. Existing methods typically guide selection using a single metric, although benchmarks often assess models along multiple dimensions. We find that model rankings are often weakly correlated across metrics, indicating that different metrics capture distinct aspects of model performance. Samples selected for one metric do not reliably preserve scores and rankings on others when used for compressed evaluation. At 80\% compression, each of the five single-metric baselines yields higher reconstruction error than equal-size random sampling in over 70\% of cross-metric comparisons. To address this challenge, we present HoloCosm, which jointly minimizes normalized score-reconstruction error across source models and metrics while constraining ranking distortion. We evaluate HoloCosm across multiple compression rates on 15 language, multimodal, VLA, and agent benchmark configurations with 85 component metrics. With 80\% compression, HoloCosm outperforms Random, the strongest baseline overall: average NMAE drops by 35.3\% to 0.236, worst-component NMAE drops by 39.5\% to 0.330, and average Spearman correlation rises from 0.884 to 0.915. In addition, HoloCosm remains competitive with existing baselines in single-metric evaluation. We will release our code and experimental results.