Game logic benchmark tests coding agents on detailed gameplay rules

GameLogicBench: Evaluating Coding Agents on Runtime Game Logic with Tick-Level State Assertions

Software EngineeringArtificial IntelligenceComputation and Language

Summary

It can be hard to check if computer programs that write game rules are doing everything right while the game is running. The authors created a new test called GameLogicBench that looks at many parts of a game at every small step to see if it follows the rules correctly. Their test includes many different game challenges and can tell if the rules are broken at any point, even if the game ends okay. They found that current coding models still struggle with bigger, more complex game tasks. Their test also makes sure agents don’t just copy public code to pass the tests.

What this means in practice

  • For game developers: Check how well AI coding tools create and maintain correct game rules across complex projects during development.
  • For ai tool builders: Use the benchmark to improve language models' abilities to write consistent and correct game logic in multiple scenarios.

Authors

Xinyu Che, Yunfei Ge, Shihao Li, Yanchen Liu, Hang Yan, Xinping Lei, Yanghai Wang, Zixuan Dong, Yifan Yao, Qianqian Xie, Letian Zhu, Jiaheng Liu

Abstract

Coding agents can modify and test code across large software projects. Game development is a domain where agents must implement gameplay rules. A game can end in a valid state even after violating its rules during the run. Current game-development benchmarks replay fixed examples, score videos, or ask another model to judge the result. However, no existing benchmark checks game rules throughout execution across varied evaluator-selected scenarios while ensuring exactly reproducible verdicts. We introduce GameLogicBench, a benchmark of 72 gameplay-logic tasks in Godot projects. An automated evaluator checks each game's rules at every simulation tick. Across 403 hand-designed scenarios, seeded parameter variations produce 1,451 test cases. To ensure that the evaluator measures behavior rather than implementation choice, it must accept different correct implementations for each task while rejecting mutants, implementations with one required capability removed. The tasks span isolated mechanics, multi-system interactions, and repository-scale features. Across 20 combinations of language models and scaffolds, the best observed run solves 52.78% of tasks. Under Claude Code, all twelve models solve fewer tasks as task scope expands from isolated mechanics, through interacting systems, to repository-scale features. Agents inspect code more often and make more tool calls on repository-scale tasks than on isolated-mechanic tasks. Most unsuccessful submissions are runnable, but implement some required game behavior incorrectly. We compared versions of our benchmark evaluator built with and without validation using mutants. Without this validation, incorrect agent submissions passed. A separate analysis finds agents copying code from public repositories when network access is open. Reliable evaluation thus depends both on what the tests reject and on what external code agents can access.