System-1 model shows mixed accuracy on biosecurity multiple-choice tasks
Auditing System-1 Models on Biosecurity-Relevant Benchmarks: Calibration, Selective Prediction, and Permutation Instability in a Non-Generative Model
Machine Learning
Summary
Some AI models called System-1 give quick answers with probabilities, making them cheaper to use than slower models. The authors tested one such model on questions related to biosecurity, like weapons detection, checking how accurate and reliable its answers were. They found its accuracy depended a lot on the task, and that it sometimes changed answers based on the order of options given. By averaging results over different answer orders, they improved accuracy, especially when the model was unsure.
What this means in practice
- •For biosecurity teams: Use System-1 models to quickly screen multiple-choice biosecurity tasks while managing confidence-based filtering to improve accuracy.
- •For automated quality control teams: Improve prediction reliability by averaging answers across different option orders for models with order-sensitive outputs.
Authors
Kimon Antonios Provatas, Ilias Georgakopoulos-Soares
Abstract
Non-generative "System-1" models return structured probabilistic decisions in a single forward pass, without autoregressive decoding, at a small fraction of the inference cost of a generative model. This makes them of interest as inexpensive components in larger pipelines, but their reliability on biosecurity-relevant tasks has not been systematically examined. We audit one commercial System-1 model on 6,020 multiple-choice items drawn from the Weapons of Mass Destruction Proxy (WMDP), a paraphrase-robust WMDP-Bio variant, and six LAB-Bench subtasks, measuring accuracy, calibration, error detection, selective prediction, and sensitivity to the order in which answer options are presented. Accuracy is strongly task-dependent. Once the vendor's uncertainty field is correctly interpreted, the model is reasonably well calibrated (pooled expected calibration error 0.034) and its top-1 probability separates correct from incorrect predictions (pooled AUROC 0.820), though both degrade substantially on the weaker tasks. Under four cyclic rotations of the answer options, 37.4% of WMDP-Cyber items receive different answers; a control using byte-identical repeated calls attributes most of this to option order rather than run-to-run variation. Averaging probabilities across rotations improves WMDP-Cyber accuracy by 3.8 percentage points, and applying it only to low-confidence items recovers most of that gain at well under the cost of averaging every item.