Opti-Agent-Bench: Benchmarking End-to-End Optimization R&D Agents on Real-World Business Problems
2026-07-12 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors created Opti-Agent-Bench, a new test that checks how well large language models (LLMs) can handle the full process of solving complex optimization problems, starting from understanding business needs to making models, writing code, and explaining results. Their benchmark is designed to be realistic and avoids easy pattern-matching tricks, testing the model’s true understanding step-by-step. They found that current LLMs often miss important parts of the problem, make inconsistent models and code, and produce unclear reports, problems that simpler tests don't catch. This shows the need for better evaluation methods to improve these AI tools.
Large Language ModelsOptimization ProblemsMathematical ModelingInteger ProgrammingRobust OptimizationStochastic ProgrammingNon-convex OptimizationBenchmarkingModel-Code ConsistencyReport Generation
Authors
Yongchang Fu, Xinjie Huang, Chengjun Dai, Chengzhe Feng, Junshao Zhang, Hong Zhu
Abstract
LLM-based agents are increasingly deployed to solve optimization problems, yet existing benchmarks evaluate them on pre-structured mathematical formulations that bypass the most critical challenge: translating complex business requirements into correct models and solve efficiently. We introduce Opti-Agent-Bench, an end-to-end benchmark that evaluates Large Language Models (LLMs) across the complete optimization R&D pipeline, from understanding business-language descriptions through mathematical modeling, algorithm selection, and code implementation, to solution report generation. Our design rests on three pillars: (1) businesssemantic authenticity with anti-template traps that defeat pattern matching; (2) modular evaluation with cross-module consistency checking across Problem Understanding, Formal Modeling, Implementation, and Reporting; and (3) the ORAC bi-level validity framework that simultaneously ensures task quality and scoring integrity. Across several industrialscale tasks spanning integer programming, robust optimization, stochastic programming, and non-convex optimization, we expose critical failure modes of current models, including constraint omission, model-code inconsistency, and report-implementation divergence, that remain invisible under conventional single-metric evaluation.