SkillEval reveals detailed skill profiles of large language models
SkillEval: Learning Interpretable Ability Profiles of LLMs via Cognitive Diagnosis Models
Computation and LanguageArtificial IntelligenceMachine Learning
Summary
Current assessments of large language models (LLMs) mostly give a single overall score, which misses the variety of different skills these models have. The authors introduce SkillEval, a method that breaks down LLM performance by specific skills using ideas from psychology to better understand how models perform. They tested SkillEval on thousands of models across many tasks and found it can predict performance more accurately than simple averages. This method also helps assign tasks to the best model for efficiency and offers new ways to evaluate and compare models.
What this means in practice
- •For ai system operators: Assign queries dynamically to models best suited by skill, reducing cost and improving response quality.
- •For model developers: Evaluate new models efficiently on skill dimensions without extensive testing using pretrained cognitive diagnosis models.
Authors
Berke Arda, Peng Cui, Qiaoyuan Zheng, Rudolf Debelak, Mubashara Akhtar, Mrinmaya Sachan
Abstract
Current evaluation of LLMs on benchmarks typically aggregates performance over individual examples into a single scalar score. This approach is overly coarse and ignores the diverse scope of skills covered by benchmark datasets. To address this limitation and better exploit benchmark data, we propose SkillEval, a new evaluation framework that performs cross-model, skill-level analysis. The framework integrates advances from two areas: (1) it leverages the meta-cognitive capabilities of LLMs to automatically discover interpretable skills from questions, and (2) it applies cognitive diagnosis models (CDMs) from psychometrics to estimate per-skill abilities of LLMs. We conduct experiments on 3,811 LLMs evaluated on 9,523 items across five benchmarks spanning diverse domains (MATH, BBH, GPQA, MuSR, IFEval). Compared to the traditional average-based evaluation paradigm, SkillEval produces reliable, more calibrated and informative yet concise model profiles that more accurately predict model behavior on unseen items. We also demonstrate the utility of SkillEval in two downstream tasks: (1) LLM routing, which dynamically assigns each query to a capable but efficient model, reducing inference cost by over 60% while achieving improved performance, and (2) Computerized Adaptive Testing (CAT), where an unseen model can be efficiently evaluated using a trained CDM. Furthermore, we show that these skill profiles provide useful insights, for example, how benchmarks share skills and how instruction-tuned models differ from their base models.