Typed decision models struggle to reject missing correct answers in arithmetic
When the Right Answer Is Missing: An Arithmetic-Dependent Rejection Bottleneck in Jev
Machine LearningComputation and Language
Summary
Some smart computer programs pick answers from a list instead of making them up. This study found that when the right answer isn’t on the list, these programs often fail to say "none of these" and instead pick wrong answers, especially for math problems. However, when the program tries to check if an answer is right or wrong directly, it does much better. The researchers also show a simple fix that helps the program refuse wrong answers more often without needing complicated retraining.
What this means in practice
- •For ai system designers: Improve numerical decision models by implementing thresholds to better reject incorrect answers when correct ones are missing.
- •For automated customer service teams: Enhance chatbot response accuracy by reducing incorrect answer selections when no valid numerical option exists.
Authors
Jike Zhong, Ming Li, Yuxiang Lai
Abstract
Typed decision models such as Jev offer an efficient alternative to generative LLMs in decision-making workflows by selecting directly from predefined options. When candidate sets contain no valid answer, TypeSafe recommends including an "other" or "none-of-the-above" option to enable rejection. In this report, however, we identify an arithmetic-dependent rejection bottleneck: Jev reliably selects correct numerical answers when available but frequently accepts incorrect alternatives when they are absent despite an explicit rejection option. On paired arithmetic problems, answer-present accuracy reaches 99%, while correct rejection falls to 7%. Moreover, this gap persists across numerical magnitudes, operation depths, contextual formulations, and rejection labels, and extends to scenarios such as time calculation and capacity rounding. Yet native Boolean verification achieves 99% exact-match accuracy on the same answer-absent arithmetic cases, showing that categorical rejection can fail even when the model successfully verifies candidate correctness. Finally, we show that a simple decision threshold selected on separate development problems raises arithmetic rejection accuracy from 7% to 79% while retaining 97% answer-present accuracy, substantially mitigating the failure without retraining or additional inference.