Large language models help improve game algorithms with limited edits
OptiArena: Can LLMs Improve Executable Algorithms under Fixed Resource Budgets?
Computation and Language
Summary
Improving computer game-playing programs by making small changes can be tricky, especially with limits on how much can be edited or tested. The authors introduce OptiArena, a system to see if AI language models can make better versions of these programs within limited steps and computing resources. They tested many models on different games and found that AI helps more with starting from very basic programs than making small improvements to already good ones. This platform helps explore how AI can optimize code when resources are tight.
What this means in practice
- •For software developers: Use OptiArena to test if AI models can improve small changes to game algorithms efficiently within resource limits.
- •For ai application engineers: Evaluate the trade-offs of enhancing existing code versus starting from minimal setups when integrating AI with bounded computational budgets.
Authors
Wenjun Peng, Xinyu Wang
Abstract
Static QA and code-generation benchmarks only partially capture the role that large language models (LLMs) now play as coding agents and research tools. We introduce OptiArena, a budget-controlled testbed for studying whether LLMs can improve executable game-playing algorithms through five rounds of code edits within a fixed minimal scaffold and under bounded evaluator feedback and fixed resource budgets. The testbed uses two optimization regimes, surface obfuscation controls, calibrated references, held-out/stress splits, and diagnostics for degradation and exceptional failures, with LLM API cost reported separately from local evaluator wall-clock. The empirical study asks three questions: whether models can close the calibrated gap between a designated weak starter and an editable competent baseline, whether they can refine editable competent baselines without damaging them, and whether gains survive surface obfuscation controls. Across twelve frontier LLMs and five games, models improve designated weak starters more consistently than they refine editable competent baselines, with substantial variation across games and models. OptiArena provides a practical testbed for measuring bounded-resource algorithm optimization within the five-edit, fixed-scaffold setting studied here. Code is available at https://github.com/WJ-Peng/OptiArena.