Summary
Some AI systems give answers with probabilities, helping software decide what to do next. The authors found that these probabilities can be wrong or sensitive to small changes in wording, which can cause software to make bad decisions without clear explanations. They created TypedBench, a test set to check how well these systems perform on tasks where decisions matter, especially when costs of mistakes differ. Their tests showed current models often have problems like being too uncertain or slowing down with more options. The paper suggests evaluating such models not just by accuracy but also by how well their probabilities match reality and how stable their decisions are.
What this means in practice
- •For software developers: Test and improve AI models that produce probability-based decisions, ensuring safer and clearer automated actions in software.
- •For risk management teams: Evaluate AI outputs under different cost scenarios to better manage and understand possible risks in automated decision systems.
Authors
Rahul Sharma, Andrew B. Ducan, Gaétan Marceau Caron, Sebastian J. Vollmer
Abstract
System One models output calibrated probabilities over typed answers such as categorical choices, ordinal levels, or binary outcomes, via a non-generative interface. Software can act on these probabilities through thresholds, cost-weighted choices, and escalation rules. Consequently, if these probabilities are miscalibrated or wording-sensitive, the software ma take unintended actions leaving human operators with no textual rationale to inspect. Current evaluations largely report accuracy and calibration on public classification datasets without a clear reference. We present TypedBench, a benchmark for typed decision models like Jev built from seven policy-labelled generators and nine evaluation suites. We report accuracy as median and range across paraphrases, and calibration error relative to the finite-sample noise floor of a matched, perfectly calibrated predictor. We assess probability quality through selective prediction, ordinal proper scoring rules, and realised cost under asymmetric cost matrices. We evaluate a hosted model, an open encoder, and a family of open decoders spanning 0.8B-9B parameters on identical items. The hosted model follows the stated policy but is wording-sensitive and systematically underconfident; under asymmetric costs, using its probabilities can be worse than taking its top answer. Decoders route exactly and are slow as options or questions are added. The decoder is least accurate on policy questions and degrades with more options. Overall, typed decision models must be evaluated jointly on policy adherence, wording robustness, probability quality, and induced decision outcomes.