Nlpdata2opt measures llm skills for real optimization problem setup
NLData2Opt: Benchmarking Parameter Derivation and Prediction for Modeling Optimization Problems from Natural Language and Data
Computation and Language
Summary
Turning real-world descriptions and messy data into math problems for optimization takes time and is tricky. The authors created NLData2Opt, a test that challenges AI language models to find useful numbers from confusing data, calculate missing details, and predict future values based on past info. They found that current AI models do much worse on this test compared to simpler ones, showing that earlier tests made these models look better than they really are at this task. This new benchmark helps developers understand where AI still needs improvement for practical optimization modeling.
What this means in practice
- •For data science teams: Evaluate AI tools on their true ability to prepare optimization inputs from incomplete and noisy data.
- •For supply chain planners: Use AI models tested by NLData2Opt for more reliable extraction and prediction of parameters to support operational decision making.
Authors
Lijia Liu, Kyohei Atarashi, Jiyi Li, Koh Takeuchi, Shunji Umetani, Hisashi Kashima
Abstract
Formulating optimization problems from raw inputs of problem descriptions and data is time-consuming and costly. While Large Language Models (LLMs) offer a promising way to automate this process, existing benchmarks overestimate their capabilities by relying on idealized inputs where all data is relevant and complete. This diverges from reality, where data-centric reasoning is required to extract, compute, and predict parameters from noisy sources. To address this, we introduce NLData2Opt, a benchmark to evaluate LLMs on the multi-stage reasoning inherent in practical optimization workflows. NLData2Opt decomposes problems into problem descriptions and data, and requires models to (1) extract relevant parameters from data containing misleading attributes, (2) compute missing parameters derived by transforming existing data attributes, and (3) predict unavailable future values from historical observations. Evaluation results show substantial performance decline relative to existing benchmarks, indicating that benchmark progress does not directly translate to formulation accuracy in practical optimization workflows. NLData2Opt is available at https://github.com/sumarivyn/NLData2Opt.