Accuracy Hides How Language Models Fail: Measuring Failure States Under Matched Output Budgets
2026-07-27 • Computation and Language
Computation and Language
AI summaryⓘ
The authors point out that usual language model tests mix up two things: whether a model actually finishes giving a usable answer, and whether that answer is correct. They suggest a new way to evaluate models by first checking if the answer is complete and properly formed, then separately checking if it's right. They tested this on several models and found that finishing the answer and judging correctness can vary a lot depending on how models are run and checked. Their work shows that just giving an accuracy number misses important details about how the answers were generated and verified.
language model evaluationaccuracyterminationanswer parseabilityverification coverageQwen modelDeepSeek modelMATH benchmarkARC-Challenge benchmarktest-time methods
Authors
Zongyou Yang, Yinghan Hou
Abstract
Language-model benchmarks collapse two distinct measurement questions into a single accuracy score: whether a response reached an evaluable state, and whether its answer was judged correct. We introduce a two-layer evaluation framework that separates scorer-independent execution evidence, including termination, answer exposure, parseability, and completion length, from scorer-dependent correctness. Across 2,550 outputs from five fixed Qwen and DeepSeek configurations on MATH and ARC-Challenge, matched 2,048-token limits produce sharply different execution mixtures: 49 of 450 Qwen MATH outputs terminate without a final answer, compared with 5 of 300 DeepSeek MATH outputs and none of the 750 ARC outputs. Among the same 300 DeepSeek MATH question-model pairs, no missing-final length termination is observed at 8,192 tokens. A coverage-audited targeted verification study further shows that candidate-selection and aggregation policies can substantially alter comparative accuracy estimates. These results demonstrate that accuracy conflates execution case mix with verification policy. Evaluations of test-time methods should therefore report pre-intervention execution states, verification coverage, and scorer provenance alongside accuracy.