ORSMT improves checking and fixing math models in AI optimization tools
ORSMT: Unleashing the Potential of Satisfiability Modulo Theories for Semantic Verification and Repair in LLM-Based Optimization
Artificial IntelligenceSoftware Engineering
Summary
Optimization problems often come with several different versions, like written math, computer code, and reports. These versions can look different but still mean the same thing, or sometimes they might miss or mix up parts. The authors created ORSMT, a tool that carefully compares these versions to check if they really match in meaning. ORSMT can find where differences come from and even fix mismatches, making the AI-generated optimization results more reliable. Testing on thousands of problems showed ORSMT boosts agreement between model descriptions and code, and it can fix nearly all errors it finds.
What this means in practice
- •For optimization software developers: Ensure AI-generated math models and solver code align correctly by automatically verifying and fixing inconsistencies in optimization pipelines.
- •For business analytics teams: Improve trust in AI-driven optimization solutions by detecting and repairing mismatches between natural language problem descriptions and generated solver code.
Authors
Rui Zhang, Chenyu Zhou, Jianghao Lin, Yaoxin Wu, Dongdong Ge, Wen Song
Abstract
Large language models increasingly automate optimization modeling from natural language problems, producing written mathematics, solver code and a returned report. Verifying one optimization problem across these artifacts requires semantic comparison: unequal models can attain the same objective, equivalent models can use different auxiliary variables, and mutually consistent artifacts can omit the same source requirement. ORSMT provides a typed semantic interface and exact SMT relation queries. It aligns observable decisions, projects auxiliary variables, proves or refutes feasible region and objective relations, localizes refutations to implicated rows or domains and links source texts when available. It then reconstructs one artifact from another, and each adopted candidate passes the required relation checks. Across 3,034 problems and six generator sizes, ORSMT improves the model-code match rate in all 72 benchmark and generator combinations before selection. Protected selection raises the aggregate rate from 43.2% to 88.2% and the share that are both correct and matched from 25.2% to 40.2%. On the 1,905 refuted outputs, certified voting under the shared fallback makes 50.9% of outputs correct and matched, compared with 44.8% for ReLoop and 49.4% for Opt-Verifier. A separate one-round source revision followed by exact reconstruction restores a model-code match for 84.4% of the refuted pairs.