Large language model plans flexible evaluations for complex optimizations
LLM-Based Evaluation Planning for Heterogeneous Optimization
Machine Learning
Summary
Some optimization problems need choices about which solutions and tests to run because tests have different costs and reliability. The authors propose using a large language model (LLM) to make these decisions instead of fixed human-designed rules. Their method, called REP-HO, takes information about the current problem state and decides where to spend resources adaptively. Experiments show this LLM-based planner performs as well as or better than traditional methods and adjusts plans depending on constraints and budgets. This suggests that LLMs can be useful for making smarter, more flexible plans in complicated optimization tasks.
What this means in practice
- •For optimization engineers: Allocate testing resources more effectively during complex multi-objective optimization with varying constraints and costs using an LLM-based planner.
- •For supply chain analysts: Adaptively decide which performance metrics and constraints to evaluate under cost limits in supply chain optimization problems.
Authors
Nian Ran, Sebastian Brulin, Markus Olhofer, Mauricio A. Álvarez, Richard Allmendinger
Abstract
Heterogeneous optimization problems require an optimizer to decide not only which candidate solutions to evaluate, but also which objectives and constraints should be evaluated under different costs and fidelities. Existing SOTA methods such as MFE-NSGA-III address this setting with manually designed selection metrics, but their behavior is largely determined by predefined heuristic rules and hyperparameters. In this work, we propose REP-HO, a reasoning-based evaluation planner that replaces the rule-based mixed-fidelity selection step with a large language model (LLM) planner. The LLM receives the same primitive optimization-state information as the heuristic planner, such as surrogate predictions, uncertainties, evaluation costs, and budget status. We first show that REP-HO achieves competitive performance compared to the human-designed metric-based method (MFE-NSGA-III). We then further enhance each decision with an adaptive nearest-neighbor summary, leading to better and more stable results. We evaluate REP-HO across five heterogeneous regimes covering cost imbalance, constraint difficulty, surrogate fidelity, budget level, and objective conflict. Across these regimes, one of the two variants obtains the best mean ranking score in every regime, REP-HO(Deepseek) is first or second throughout, and the planner shows adaptive allocation behavior, such as prioritizing constraint evaluations under tighter budgets and reallocating constraint evaluation budget when surrogate reliability changes. These results suggest that LLMs can serve as flexible evaluation planners for heterogeneous optimization, offering a general alternative to fixed hand-designed metrics.