Large language models struggle with complex rule reasoning tasks
RuleTrek: Benchmarking Agentic Rule Reasoning in Complex Scenarios
Artificial Intelligence
Summary
Computers that can understand and use language often find it hard to follow complicated rules spread across many documents. The authors created RuleTrek, a new test to see how well these computer programs can identify and apply rules in detailed and tricky situations. They tested 11 popular language models and found that none could reliably understand and use the rules correctly. This shows there is still a lot to improve for computers trying to solve complex rule-based problems.
What this means in practice
- •For software developers: Evaluate and improve AI systems' ability to process complex rules across multiple documents using the RuleTrek benchmark.
- •For compliance teams: Test AI tools for correctly interpreting intricate policy or regulation rules spread across various files before deployment.
Authors
Chenhao Yuan, Bohan Yu, Yinhao Xu, Chao Yin, Chenxi Zhou, Zhaoxuan Wang, Shuwen Xu, Shaoping Huang, Haolin Ren, Jiaxiang Liu, Yao Xu, Jun Zhao, Kang Liu
Abstract
Large language models (LLMs) increasingly tackle complex real-world tasks in specialized domains. However, identifying and correctly applying domain-specific rules remains challenging when scenario information is distributed across multiple files. Existing rule-reasoning benchmarks typically represent scenarios using only a few factual statements, simplifying the identification and application of relevant rules. To address this gap, this paper introduces \textbf{RuleTrek}, a multi-domain benchmark for agentic rule reasoning in complex scenarios. Its rule-grounded construction pipeline extracts structured rule graphs, generates and incrementally composes rule-specific sub-scenarios while ensuring consistency and solvability. To systematically evaluate agentic rule reasoning in complex scenarios, RuleTrek assesses three capabilities in interactive workspaces: Rule Identification (RI), Rule Deduction (RD), and End-to-End Rule Reasoning (E2E). Experiments on 11 frontier LLMs show that the highest RI and RD scores are only 62.47\% and 60.43\%, respectively, while E2E scores range from 26.97\% to 55.29\%. These results demonstrate that current LLM agents face substantial challenges in rule reasoning over complex scenarios. We make our code and dataset publicly available at \href{https://github.com/yuan247/RuleTrek}{RuleTrek}.