LLM agents tackle real-world operations research problems with some success
OR-License: Evaluating LLM Agents as Real-World Operations Research Practitioners
Computation and LanguageArtificial Intelligence
Summary
Solving complex business problems often involves turning plain descriptions into mathematical or computer models to find the best solution. The authors created OR-License, a test set using real company projects to see if language AI agents can solve these problems directly from business documents, without human experts rewriting them first. Some AI models came close to expert solutions but still had trouble guaranteeing fully feasible and reliable answers. This shows AI can help, but it needs to improve understanding and reasoning to fully replace human experts in operations research tasks.
What this means in practice
- •For industrial operations teams: Evaluate AI agents to assist or automate operational planning tasks using original business data without expert reformulation.
- •For software developers for optimization tools: Incorporate AI agents into software that translate client data into formal optimization problems, improving solution development workflows.
Authors
Peisong Wang, Feixue Liu, Yize Meng, Bowen Liu, Ruoyu Chen, Gaoyang Dai, Xing He, Aiqian Li, Fengchi Yuan, Aochuan Chen, Chenyi Zi, Yuhan Li, Hongchuan Zeng, Yimo Yan, Nuo Chen, Jie Fu, Zuo-Jun Max Shen, Yong-Hong KUO, Jia Li
Abstract
Can large language model (LLM) agents deliver expert-level operations research (OR) solutions directly from original client materials, without prior expert reformulation? Existing OR benchmarks typically begin with self-contained descriptions or predefined optimization tasks, underrepresenting the translation of original business materials into professionally validated solutions. We introduce OR-License, an end-to-end benchmark of 31 instances across 10 industrial scenarios from paid OR engagements totaling USD 3.68 million. It preserves original requirements, input data, deliverable formats, and information boundaries after de-identification. To study information acquisition, we introduce ORSim, a controlled client simulator that discloses business facts without supplying OR expertise. Hidden evaluation checks feasibility and compares objective quality against original expert solutions accepted by clients before benchmark construction. Deployment-Adjusted Quality (DAQ) averages expert-relative quality across instances, assigning zero to infeasible deliveries. Across 11 frontier LLMs using coding-agent harnesses, several exceed the expert reference in conditional objective quality, but none matches its DAQ; the strongest primary Claude Code configuration achieves 88.39% strict feasibility and a DAQ of 0.87 against the expert reference of 1. Client information acquisition extends beyond clarification into solution development and validation, with some later stages showing higher confirmed issue-resolution proportions. Trajectory analysis identifies formulation and implementation failures, while task-equalized research-chain completion rates range from 24% to 45% across models on evaluable tasks. These findings motivate sustained information-gap resolution, faithful problem formulation, and evidence-linked self-improvement for reliable long-horizon agents.