Prediction-powered methods improve ai evaluation accuracy across domains
Prediction-Powered Smoothing and Validation for Disaggregated AI Evaluation
Artificial IntelligenceMachine Learning
Summary
Measuring how well AI performs in different areas is hard because testing everything takes too much time and effort. The authors propose new methods that use smart predictions and statistical smoothing to get better and more reliable estimates of AI performance in different categories, even when data is limited. They also created a way to check which method works best without needing extra data. Their experiments show these methods give more accurate results and better confidence in evaluations.
What this means in practice
- •For machine learning engineers: Improve reliability of AI system performance metrics across varied tasks using prediction-powered smoothing and validation techniques.
- •For quality assurance teams: Assess deployed AI agents more accurately with limited labeled data by combining domain-specific and collective estimates.
Authors
Sho Kawano, Zehang Richard Li, Paul A. Parker
Abstract
Evaluating an AI system requires disaggregated assessment, as performance varies across domains such as benchmark task types or conversation types in deployed agents. Exhaustive testing is expensive, so evaluation rests on a sample of labeled units. We treat the evaluation set as a finite population and seek accurate point and interval estimates of each domain mean. Direct estimators, including prediction-powered inference (PPI), use only a domain's own labels and are imprecise where labels are few. Small area estimation addresses this problem, and we build on it to develop an integrated workflow for estimation and validation. For estimation, we propose prediction-powered smoothing (PP-S), a Bayesian model fit to each domain's prediction-powered estimate, with an extension that borrows strength across a reporting taxonomy (PP-TS). For validation, we derive a new, approximately unbiased design-based cross-validation score for choosing among direct and smoothed estimators. We study a curated benchmark with verifiable grading and deployed agent traffic graded by humans, each with every outcome observed. In both, the proposed estimators improve on the direct estimators in point and interval estimation, with near-nominal coverage. At the same sampling budget, our score selects as well as an independent validation sample does and estimates the selected estimator's error far more accurately.