User bias shapes data and scores in common AI benchmarks
Whose Data, Whose Benchmark? A User-Centric Framework for Auditing Evaluation Resources
Computation and LanguageArtificial IntelligenceComputers and SocietyEmerging TechnologiesHuman-Computer Interaction
Summary
Benchmarks are tests used to measure how well AI models perform, but the people who make these tests often come from specific places or backgrounds. The authors created a framework that checks whose perspectives and needs are built into these tests and whose they represent. They found many benchmarks focus mainly on American norms and that evaluation scores can change a lot depending on how answers are judged. Their work helps make clear what the results of these tests really mean by showing whose views are included or left out.
What this means in practice
- •For ai development teams: Audit and adjust benchmarks to better reflect diverse user needs and reduce geographic or cultural bias in model evaluation.
- •For dataset curators: Use the framework to improve benchmark documentation and scoring to clarify assumptions about target users and evaluation goals.
Authors
Mir Tafseer Nayeem, Davood Rafiei
Abstract
Benchmarks shape how model capabilities are measured and interpreted, yet their scores can obscure whose conventions, contexts, and judgments define successful performance. We introduce a user-centric audit framework that distinguishes users who contribute benchmark content from those whose needs evaluation results are intended to represent. The framework operationalizes seven dimensions across benchmark content, metadata, documentation, and scoring procedures. Across 16 resources, we find that American-centered conventions account for 62.4-94.4% of matched signals in 11 text benchmarks, while geographic visibility in five image-text resources is concentrated and strongly associated with national GDP share (mean Pearson r = 0.780; Spearman p = 0.778). Documentation reveals a systematic source-target gap: all 16 resources operationalize source-user context, but none operationalize target-user context. Paired scoring experiments further show that meaning-preserving answer variants reduce DROP exact-match accuracy from 100.0% to 33.3%, while TruthfulQA exhibits metric-dependent reductions in reference similarity. Our framework and documentation checklist make user assumptions auditable, clarifying what benchmark scores represent and where evaluation claims are supported.