Tabular foundation models show mixed results in expensive optimization
Benchmarking Tabular Foundation Models as Surrogates in Expensive Evolutionary Optimization
Neural and Evolutionary ComputingMachine Learning
Summary
Optimizing complex problems can be very costly because each attempt to find a better solution takes a lot of resources. The authors studied a new type of model called TabPFN to see if it can predict good solutions and reduce those costs. They found that while TabPFN works well in some cases, it doesn't always outperform traditional methods. Its success depends on the problem type, data available, and how it is used within the optimization process. The authors suggest carefully choosing when and how to use TabPFN to get the best results.
What this means in practice
- •For engineering design teams: Assist in selecting surrogate models for costly optimization tasks to improve design efficiency based on problem traits and data availability.
- •For industrial process optimizers: Guide use of TabPFN for reducing expensive evaluations when tuning multi-objective or constrained processes, improving resource use.
Authors
Lu Han, Jin Wang, Yuchen Li, Haoran Gu, Shulei Liu, Ziyang Shi, Wenao Lu, Handing Wang
Abstract
Surrogate-assisted evolutionary algorithms (SAEAs) are effective methods for solving expensive optimization problems (EOPs), where surrogate models replace most expensive evaluations and critically influence the final optimization results. In recent years, tabular foundation models have advanced rapidly, and the Tabular Prior-data Fitted Network (TabPFN) has been adopted as a surrogate model for EOPs due to its strong predictive capability, demonstrating promising performance. Motivated by its potential as a surrogate model in SAEAs, this work conducts a comprehensive study that combines extensive experiments with in-depth theoretical analysis to investigate the effectiveness of TabPFN. Specifically, we perform experiments across both offline and online SAEA settings, covering diverse problem scenarios such as single-objective, multi-objective, constrained, combinatorial, mixed-variable, and engineering optimization problems. In addition, we further analyze the advantages and limitations of TabPFN within SAEAs and provide practical guidelines for its application in different optimization settings. Results show that the effectiveness of TabPFN is highly problem dependent, and it cannot replace conventional surrogates universally. Overall, TabPFN should be adopted selectively according to data availability, landscape complexity, search space characteristics, and its role within the algorithm. Customized model management strategies and role-specific algorithm design are necessary to fully exploit its advantages and avoid its pitfalls.