Generative model evaluation improved by sparse and unpaired query matching
Benchmark Prediction with Sparse and Unpaired Cached Responses
Machine Learning
Summary
Testing AI models on every question from every test is very slow and expensive. The authors introduce a new way called Product Kernel Perspective Space (PKPS) to compare AI model answers even when tests only cover some questions for some models. This method uses similarities between different questions, not just identical ones, making it faster and requiring fewer examples. Their approach shows better results when predicting how AI models will perform across many tasks.
What this means in practice
- •For machine learning engineers: Predict model performance on incomplete benchmark data to save evaluation time and resources.
- •For ai testing teams: Use PKPS to accurately compare different AI models when full benchmark data is unavailable or costly to obtain.
Authors
Hayden Helm, Avijit Ghosh, Ben Johnson, Carey Priebe
Abstract
Evaluating a generative model on all queries from all available benchmark tasks is prohibitively expensive. As such, models are often evaluated on different subsets of tasks and/or different subsets of queries from a given task. The resulting collection of responses and the corresponding (model, task) score matrix are sparse and unpaired. In this paper, we introduce the Product Kernel Perspective Space (PKPS), a collection of low-dimensional representations of models based on their embedded responses. PKPS generalizes existing methods for representing black-box generative models by comparing responses to similar -- rather than just identical -- queries and is therefore able to utilize sparse and unpaired information. Theoretically, we show that benchmark prediction using PKPS is query-efficient relative to methods that only use information from dense and paired portions of the evaluation data. Empirically, we show that PKPS-based methods outperform baselines on benchmark prediction across two large benchmark suites.