Vision language models improve complex optimization problem solutions
VLM Fine-Tuning for End-to-End Combinatorial Optimization
Computation and Language
Summary
Combinatorial optimization involves solving problems by finding the best arrangement or path, like planning routes or job schedules. The paper shows that large language models can do better on these problems by including pictures that represent the problem alongside text descriptions. These visual clues help the model understand connections and layouts that text alone misses. The gains are bigger for harder problems and larger cases.
What this means in practice
- •For logistics planners: Improve routing solutions for large vehicle fleets by using combined visual and textual problem inputs to generate better plans.
- •For manufacturing schedulers: Generate higher quality job shop scheduling solutions by incorporating visual representations into AI solvers.
Authors
Qingsong Yan, Xia Jiang, Yaoxin Wu, Wen Song, Lu Zhang, Yingjie Zhou
Abstract
Large language models (LLMs) have provided a unified interface for end-to-end combinatorial optimization (CO), but textual serialization alone may obscure spatial and relational structures that are important for generating effective CO solutions. This paper presents a general-purpose vision-language solver that augments textual instance descriptions with input-derived visual representations. A single vision-language model (VLM) is applied across different CO tasks and trained using supervised fine-tuning followed by verifier-guided reinforcement learning. While the visual inputs contain no gold solutions or solution-derived information, our experiments show that the VLM generally improves solution quality over its text-only counterpart, with particularly clear gains on more complex CO problems such as CVRP and JSSP. The advantage of visual information is more pronounced at large problem scales.