Typed decision models show varied accuracy detecting missing answers
Benchmarking Candidate Coverage in Typed Decision Models
Artificial IntelligenceComputation and Language
Summary
This paper looks at how typed decision models handle answer choices when some options might be missing. The authors found that simply checking how accurate a model is when all answer choices are present doesn't tell the full story. They tested two models, Laya and Jev, on different datasets and measured how well they detect missing answers and avoid wrongly rejecting valid options. Their tests show different strengths and weaknesses between the models and suggest multiple ways are needed to measure model performance properly.
What this means in practice
- •For natural language processing engineers: Evaluate decision models to better detect when reference answers are missing in text classification tasks.
- •For ai model deployment teams: Improve rejection policies in AI systems by separately measuring coverage and rejection performance using the benchmark protocol.
Authors
Jiawen Lu, Tongtong Wu
Abstract
Typed decision models return choices or distributions over answer options supplied at request time. Accuracy with complete options does not establish whether a model recognizes that a reference answer is missing or avoids rejecting valid candidates. We present a paired candidate-coverage benchmark protocol and an initial evaluation of Laya and Jev across AG News, DBpedia, Emotion, and TREC. The models receive identical frozen texts and requests: 300 calibration and 589 test texts yield 23,932 predictions per model. Present/absent pairs match ordinary candidate count, and name variants preserve descriptions, members, and order. Native rejection behavior differs sharply: at five TREC candidates with natural names, Laya detects 97.2% of missing-answer cases but falsely rejects 69.7% of present controls; Jev's rates are 24.8% and 0.0%. Calibration-only none-score thresholds change these rates to 33.9%/3.7% and 45.0%/1.8%, respectively. On DBpedia, Jev's high coverage-score AUROC supports a stronger operating point, whereas both models have weak complete-set accuracy on Emotion. Competence-conditioned analysis, probability-precision sensitivity, and interface audits show why classification, score ranking, and rejection policies need separate measurement. This initial benchmark is descriptive and limited to reference-label omission; it does not establish natural out-of-scope generalization, causal mechanisms, or a new rejection method.