AI summaryⓘ
The authors tested how making evaluations faster and cheaper affects the conclusions we draw about AI model behavior, focusing on responsible AI benchmarks. They looked at different ways to speed up evaluation, like batching inputs, reducing number size (quantization), and using smaller test sets, comparing results to a standard baseline. They found that some methods, like batching and certain quantizations, keep accuracy and bias measures stable, but others can cause notable changes depending on the model and test details. Their work shows that making evaluation more efficient can change what we conclude about AI models, so these methods need careful checking. The authors provide their data and tools online for others to explore these effects.
responsible AIbenchmarkingmodel evaluationbatchingquantizationmixture-of-expertsaccuracybiassubset stabilityenergy consumption
Authors
Ahmed El Kady, Aravind Narayanan, Rehana Noorani, Yani Ioannou, Shaina Raza
Abstract
Efficient evaluation changes the protocol used to support claims about model behavior, yet it is rarely tested whether those claims remain stable after the evaluation itself is made cheaper. We stress-test conclusion robustness in responsible-AI benchmarking by evaluating three dense and mixture-of-experts models on BBQ and BBQ-V under seven conditions spanning batching, quantization, benchmark reduction, and their combinations. Rather than treating preserved aggregate accuracy as sufficient, we compare accuracy, bias severity and prevalence, reasoning quality, subgroup behavior, subset-membership stability, runtime, and measured GPU energy against a full-benchmark BF16 baseline. Larger batching keeps accuracy within 0.35 percentage points of baseline and produces comparatively small subgroup changes, while reducing energy in five of six model--dataset settings. INT8 largely preserves quality but uses 1.79--4.26$\times$ baseline energy. INT4 causes larger, model- and context-dependent changes. Reduced benchmarks provide the most consistent savings, but very small subsets are substantially more sensitive to which items are retained. Efficient evaluation should therefore be treated as a measurement intervention whose validity must be checked across the conclusions the benchmark is intended to support. Our project website is https://vectorinstitute.github.io/sustainable-rai-evaluation/ and the code is available at https://github.com/VectorInstitute/sustainable-rai-evaluation.