Llm framework improves business optimization model accuracy locally
OR for AI That Does OR: Routing LLMs up the Escalator inside the OSCAR Framework
Artificial Intelligence
Summary
Translating a business problem into a math model that computers can solve is tricky because errors can lead to wrong solutions. The authors created OSCAR, a system that uses multiple small language AI models to check and improve these math models against real examples, making the results more reliable. OSCAR figures out which AI to use and when to stop trying, balancing cost and accuracy. This helps companies use AI locally and securely while keeping their business rules clear and consistent.
What this means in practice
- •For business operations teams: Generate accurate optimization models from business rules repeatedly, minimizing costly errors and improving plan quality locally using small language models.
- •For supply chain managers: Refine routing and planning models with a cost-effective LLM framework that checks solutions against actual business constraints to improve decision quality.
Authors
Jinzhi Bu, Haixin Tang, Huanan Zhang
Abstract
Large language models can translate business descriptions into optimization models, but executable code may misrepresent constraints or objectives. A solver can then return an optimal solution to the wrong problem. Even when the solution satisfies the intended operating rules, a better plan may exist. For organizations that repeatedly use optimization modeling, an LLM-based framework should produce accurate formulations at low cost and, ideally, run locally. We study how to verify improvements and allocate attempts across LLMs that differ in price and capability. We develop OSCAR (Optimization modeling by Simulator, Coder, And Reviewer), which uses an offline Simulator certified against labeled decision examples to compare candidates and continues searching beyond feasibility. We model the search for the next certified improvement as sequential decisions under unobserved difficulty: which LLMs to call and when to stop. In a simplified known-prior setting, we give conditions under which cost-ordered escalation is optimal. For general menus, we derive a prior-free competitive guarantee. On five benchmark problems, OSCAR achieves 95% to 100% accuracy at the reported settings using two small open-weight LLMs, each deployable locally on a single GPU. Their single-attempt accuracies average 29% and 48%. In five runs per problem, Codex and Claude Code incur average token costs 3.1 and 5.8 times OSCAR's, respectively. OSCAR supports open-weight models locally or in the cloud, depending on budget and confidentiality requirements. Firms should maintain labeled decision examples of feasible and infeasible decisions to clarify plain-language operating rules. OSCAR follows these labels when an LLM's interpretation conflicts with them. As LLM capabilities and prices change, OSCAR's simple operating rules and adjustable settings help firms adapt their model choices and benefit from these advances.