Pass at k metric misses key diversity and ability changes after training
What pass@k Cannot Measure: Evaluating Diversity and Capability Retention after Post-Training
Machine Learning
Summary
pass@k measures how often a model solves problems within k tries but ignores how diverse those solutions are. The authors found that some fine-tuning methods changed the variety of answers a model produced without affecting pass@k scores. This means pass@k can overlook important changes in how capable or diverse a model’s outputs are. Their work highlights that relying only on pass@k might miss whether a model has truly improved in meaningful ways.
What this means in practice
- •For machine learning engineers: Detect deeper changes in model output variety after fine-tuning when pass@k scores are unchanged.
- •For ai system evaluators: Improve evaluation protocols by including diversity metrics beyond pass@k to better assess model improvements.
Authors
Subham Rath, Raj Dandekar, Rajat Dandekar, Sreedath Panat
Abstract
pass@$k$, the fraction of problems a model solves within $k$ sampled attempts, is the field's default protocol for deciding whether reinforcement-learning (RL) post-training on verifiable rewards improved a model. At the population level, pass@$k$ depends only on a problem's probability of a correct sample, with no term for how it is distributed across outputs. We show this gap is not academic. Training Qwen2.5-1.5B-Instruct on grade-school math with Group Relative Policy Optimization (GRPO) and with rejection-sampling fine-tuning (RFT, training on the model's own shortest verifier-passed rollout) moves three complementary diversity measures (token-level entropy, answer-level entropy, unique answers per prompt) in opposite directions, with zero overlap across three seeds per arm. The gap survives restricting to verifier-correct completions only (lexical diversity among correct solutions is 15% lower for GRPO, after controlling for length) and a count-controlled check isolating diversity among incorrect answers alone, ruling out that GRPO's higher accuracy alone explains it. Yet pass@8 and pass@32 show no consistent winner on GSM8K, and a hard MATH-500 subset shows the same pattern: separation only at low $k$. Compared against the starting checkpoint, no trained arm significantly improves hard-problem coverage: RFT is significantly worse, while GRPO is statistically indistinguishable from it - so GRPO's pass@1 edge over RFT reflects a smaller loss relative to Base, not a capability gain, a missing-control issue, not a failure of pass@$k$. On GSM8K, only pass@1, with no role in detecting diversity by construction, separates the arms cleanly, rewarding the arm whose correct solutions are least diverse. We argue this is a concrete instance of a standard evaluation protocol missing a property it is routinely used to certify.