Language model test tool generates many reasoning questions automatically
Deogen: Automatic Benchmark Generation for Reasoning over Policies
Artificial IntelligenceMachine Learning
Summary
It can be hard and expensive to create tests that check if AI models understand and follow complex rules correctly. The authors created a tool called DeoGen that automatically makes lots of these rule-following test questions by turning policies into logic trees. This way, many test cases are generated quickly, including ones where not enough information is given so the model should not guess an answer. They found that while AI models often get the right answer when enough information is available, they struggle when information is missing, even when told to admit uncertainty.
What this means in practice
- •For ai ethics teams: Create large, automated benchmarks that test AI compliance with policy rules and detect when models should abstain due to lack of information.
- •For compliance software developers: Integrate automated rule-based test generation to improve verification of AI systems in regulated environments with complex policies.$Commercial implications: Enables selling enhanced compliance testing platforms for regulated industries that require rigorous rule-following by AI models.
Authors
Sam Kouteili, William Liang, Keeran Murray, Adriana Zhou, Ramesh Viswanathan, Scott Shapiro, Ruzica Piskac
Abstract
Language models are increasingly being deployed in regulated settings, where a model must apply given rules to the facts of a situation to determine a verdict. Benchmarking this capability, known as deontic reasoning, remains a costly and manual process, with existing datasets largely hand-crafted. In this paper, we present DeoGen, a scalable, automatic, and parametrizable benchmark generator for deontic reasoning. By autoformalizing policy rules to first-order logic, we can construct decision trees that capture possible outcome trajectories. Each path down a tree becomes a benchmark entry, with each tree producing combinatorially many entries relative to tree depth. Paths ending at a leaf entail a verdict, while non-leaf paths measure a model's ability to abstain when presented with insufficient information. Evaluations show frontier models return the correct verdict 89-92% of the time. Notably, performance degrades (22-40% correct) on entries without sufficient information to claim a verdict, even with explicit instructions to abstain in such cases. In all, DeoGen presents an automatic, scalable, and parametric method for generating deontic reasoning benchmarks.