Large language models struggle with complex numerical optimization tasks
Understanding Large Language Models as Numerical Optimizers
Artificial IntelligenceMachine Learning
Summary
Large language models (LLMs) are being tested to solve math problems that involve finding the best solutions under certain rules. The authors studied how well LLMs can understand these problems and produce good answers, especially as the problems get bigger and more complex. They found that while LLMs can sometimes find near-best answers, their reliability drops with problem size and they often make errors like breaking rules or formatting mistakes. Adding a step where the model checks and improves its answers helps, but even then, mistakes remain. This means LLMs alone aren’t ready to replace traditional math-solving tools but could be combined with them.
What this means in practice
- •For optimization engineers: Use LLMs combined with numerical checks to solve medium-scale constrained optimization problems with improved reliability.
- •For software developers: Incorporate iterative refinement techniques from this study to enhance the robustness of AI-assisted mathematical problem solving tools.
Authors
Lingyun Guo, Ali Bereyhi
Abstract
Large language models (LLMs) have emerged as potential tools for numerical optimization. Despite promising results, their intrinsic optimization capabilities and limitations remain poorly understood, particularly in high dimensions. This paper investigates the ability of LLMs to exploit mathematical problem structures and generate feasible, near-optimal solutions. We develop an evaluation framework that distinguishes solution quality from execution failures, including numerical inaccuracies, constraint violations, and output-format errors. Using representative problems from convex optimization, fractional programming, and constrained projection, we assess LLM performance against deterministic reference solvers across different dimensions. Our results reveal that LLMs can capture useful optimization structures and produce near-optimal solutions, even when their outputs fail to satisfy numerical or formatting requirements. However, reliability deteriorates with increasing problem dimension and depends strongly on the underlying optimization structure. To further investigate the limitations, we deploy an iterative refinement mechanism that provides targeted feedback on feasibility and solution quality. The results demonstrate that iterative refinement substantially improves reliability and extends the range of problem dimensions over which LLMs produce valid solutions. Nevertheless, persistent numerical and structural errors limit their applicability as standalone optimizers. These findings provide insights into the capacity and limits of LLM-based optimization and motivate hybrid approaches that combine LLM reasoning with deterministic numerical verification.