Language model accuracy hides differences revealed by item analysis
Diagnosing Model--Item Interactions in Language Model Benchmarks with LSIRM
Computation and LanguageMachine Learning
Summary
Sometimes language models score similarly on tests but actually do well on different questions. The authors show how a tool called LSIRM can map where these differences happen and why, by looking at answers and question types. They found that models tend to do better or worse depending on which answer categories they face, which can change how models rank against each other. This approach helps people see hidden differences beneath overall test scores and understand how scoring choices affect which models look best.
What this means in practice
- •For machine learning engineers: Improve language model evaluations by identifying item-level performance patterns beyond overall accuracy.
- •For benchmark designers: Design better scoring methods that reveal meaningful differences in model abilities across item types.
Authors
Jinwen Luo, Han Jiang, Ziang Xiao, Susu Zhang
Abstract
Language models with similar benchmark accuracy can succeed on different items. Overall scores do not show where their performance differs or which item properties might explain those differences. We demonstrate how the latent space item response model (LSIRM) can help evaluators investigate these patterns and their consequences for model comparison. We analyze 1,348 language models on 5,761 BBH items and compare ways of presenting and scoring answers across BBH, GPQA, MMLU-Pro, and MuSR. We examine LSIRM maps, which summarize differences in performance across models and items, alongside item properties and records of correct and incorrect responses. In BBH, nearby items on these maps often share the same correct answer, such as "Yes" or "No." Across the four benchmarks, this pattern is stronger when answers are scored using option labels, such as A, B, C, and D, than when using the text of the options. We then estimate each model's tendency to perform better or worse on items with particular correct answers and quantify how this tendency contributes to its score. This separates an estimated contribution associated with correct answers from the remaining differences in item performance. Two models' total numbers of correct answers differed by only 14 out of 5,761 items. Excluding the estimated answer-category contribution from their scores moved their ranks hundreds of places in opposite directions, while keeping all other estimates unchanged. The approach helps evaluators identify differences hidden by overall accuracy and understand how the choice of what to include in a score affects model rankings.