Mmlu benchmark mainly measures memory not reasoning ability
What Does MMLU Actually Measure? A Psychometric Audit of Difficulty Structure in Aggregate Benchmark Scores
Computation and Language
Summary
MMLU is commonly used to judge how smart AI models are, but this paper shows it mostly tests how well they remember facts rather than how well they reason. The researchers found that combining reasoning and memory into one score mixes up these different skills, especially between science and non-science questions. Because of this, picking top models by overall score often misses better choices for tasks requiring strong reasoning. The authors provide a new way to analyze and report these skills separately so model evaluation can be more accurate.
What this means in practice
- •For ai model developers: Select models based on separate reasoning and fact retrieval scores to better match deployment needs.
- •For ai evaluation teams: Use the proposed framework to audit and improve benchmark tests for clearer skill measurement.
Authors
Dana Paquin, Riddhiman Jain
Abstract
Although MMLU is widely adopted as a benchmark for calibrating general AI capabilities, we psychometrically demonstrate that its aggregate score primarily evaluates a model's factual retrieval capacity rather than its reasoning ability. By calibrating item difficulty for 1,000 open-weights language models over 14,042 MMLU test items using Item Response Theory, we show that evaluating both abilities via a single test is inherently flawed. Difficulty is then regressed on a deterministic, text-extractable framework of structural complexity. Applying a joint Wald test with subject-clustered covariances demonstrates that the MMLU conflates fundamentally separable constructs. The mapping from structural complexity to difficulty is not invariant across the benchmark's STEM and non-STEM partitions. This finding has practical consequences. Aggregate leaderboard ranks track non-STEM accuracy more closely than STEM accuracy, so selecting a Top-50 model on the aggregate for a reasoning-intensive deployment displaces roughly 22% of the STEM-appropriate choices. Furthermore, when controlling for the multiple-choice guessing floor natively inside the response model, we find that higher-ability models continue to degrade more steeply under increased reasoning depth. The MMLU aggregate therefore weights retrieval capacity and reasoning stability unequally, inadvertently favoring models optimized for retrieval. We release our deterministic framework as a reproducible auditing instrument and recommend disaggregated reporting.