JOR-Bench: Japanese Operations Research Benchmarks for Large Language Models
Computation and LanguageArtificial Intelligence
Summary
The authors created JOR-Bench, a set of five Japanese tests to check how well large language models can understand and solve operations research problems, like optimization tasks. They translated existing English benchmarks into Japanese, covering over 1,300 problems in different types of programming and optimization. They tested seven language models on both English and Japanese versions and found that good multilingual models perform almost the same in both languages. However, small differences appeared, such as models sometimes misunderstanding instructions in Japanese, leading to slightly different answers. The benchmark is designed to work with any solver and helps compare model performance reliably.
Authors
Yuu Jinnai
Abstract
We present JOR-Bench, a collection of five Japanese-language benchmarks for evaluating the ability of large language models (LLMs) to formulate and solve operations research (OR) problems. Each benchmark is a Japanese translation of an existing English benchmark: IndustryOR, MAMO Complex LP, NL4OPT, OptiBench, and OptMATH, covering 1,319 problems spanning linear programming, mixed-integer programming, non-linear programming, and combinatorial optimization. JOR-Bench is a solver-independent benchmark that can be used with any solver or programming language, and consists of pairs of Japanese problem statements and expected numerical answers. We evaluate seven LLMs, including multilingual general-purpose models and Japanese-specialized models, on both the original English and the new Japanese versions, and compare performance across languages. For the main evaluation, we standardize execution with the Python interface to OR-Tools to make model outputs comparable and reproducible with open-source software. Our results show that OR formulation ability is largely language-neutral for strong multilingual models; the overall average accuracy difference between English and Japanese is only $-0.3$ pp. Yet error analysis reveals subtle cross-lingual differences, including a pragmatic disambiguation failure in some domains that causes models to output decision-variable values instead of the objective value when the prompt is in Japanese.