Large language models struggle with long manual based tasks
Tasks over Application Manuals: Revealing Gaps in Long-Horizon Procedural Reasoning for Language Models
Computation and LanguageArtificial Intelligence
Summary
Large language models (LLMs) are good at many language tasks but have trouble following very long and complex instructions from manuals. The researchers created a new test using real-world tasks from medical coding and crime sentencing, where the instructions come from huge official rulebooks. They found that current models barely get the exact answers right because they cannot reliably follow many-step procedures. This shows that existing tests might not fully capture the difficulty of real long-step reasoning tasks.
What this means in practice
- •For medical coding teams: Assess LLM tools' capabilities for assigning diagnostic codes from lengthy medical manuals before adopting them into clinical workflows.
- •For legal analytics teams: Evaluate whether language models can accurately compute sentencing outcomes from complex legal guidelines to assist legal decision tools.
Authors
Utkarsh Soni, Syed Shariyar Murtaza, Yifan Nie, Sachin Chandrasekhar, Eugene Wen
Abstract
Large language models (LLMs) have achieved strong performance on a wide range of natural language tasks, and recent benchmarks suggest that they are increasingly adept at multi-hop reasoning. However, these benchmarks are typically short-horizon, requiring only a small number of retrieval or inference steps, and provide limited evidence of reliability on real-world tasks that involve following manuals spanning hundreds of pages with complex, interdependent guidelines. In this paper, we introduce Tasks over Application Manuals (TAM), a benchmark for evaluating long-horizon procedural reasoning. We construct TAM by curating real-world tasks from two domains: ICD-10-CM clinical coding (mapping medical conditions to diagnostic codes) and U.S. federal sentencing (computing crime sentencing guideline outcomes, specifically offense levels), with human-validated labels. Each task requires following an authoritative manual with tens of thousands of rules and executing a sequence of interdependent steps across different sections to produce an exact answer. We evaluate general-purpose prompting approaches, including retrieval-augmented generation, ReAct-style prompting, and an agent-harness baseline on GPT-5, and find that the best exact-match performance remains extremely low: 1% on ICD-10-CM coding and 15.5% on sentencing tasks. These results show that current benchmarks may overestimate LLM reasoning ability and miss a key challenge: reliably following long, rule-based procedures. The complete TAM data and code are publicly available.