Game development agents struggle to meet complex design requirements
A2Z GameSpec-Bench: How Faithfully Can Coding Agents Generate Games from Game Design Specifications?
Artificial Intelligence
Summary
Making computer programs that create games exactly as planned is hard. The authors created a new test with 100 detailed game design documents to see how well AI coding agents follow these plans. They found that current AI agents have trouble meeting all the interconnected rules needed for a game to work properly. The authors also showed that giving the agents specific feedback can help improve how well they follow the designs. This work helps measure and improve AI tools that try to build games from detailed instructions.
What this means in practice
- •For game development teams: Evaluate and improve AI coding agents’ ability to generate games that match complex design specifications through structured testing and revision.
- •For software quality assurance teams: Use dependency-aware contracts and scenario-based playtesting to verify that software meets interconnected functional requirements in interactive applications.
Authors
Seonho Lee, Wonryeol Jeong, Alberto Cereser, Inha Kang, Hyeonjong Kim, Seungmin Kwak, Dongmin Park
Abstract
Delegating complete application development to coding agents requires preserving the intended design rather than simply producing plausible outputs through naive prompting. Game development provides a demanding testbed, as long-form Game Design Documents (GDDs) describe requirements that must work together across game logic, visual rendering, and player interactions. However, existing game-development benchmarks typically use compact specifications and provide limited support for evaluating interdependent requirements across these aspects in long-form GDDs. We introduce A2Z GameSpec-Bench, a benchmark of 100 long-form GDDs for evaluating end-to-end game development by agents. We measure faithfulness by checking whether the game satisfies the GDD requirements and preserves the relationships among them. Each GDD is turned into a dependency-aware contract that contains rules, constraints, and prerequisite relations. Following game-development practices, we combine source-code inspection with agent-generated test policies for scenario-based replay and adaptive playtesting. The contract remains fixed across agents and revision rounds, while judgments and evidence linked to the same requirements support consistent comparison and failure detection. Our evaluations show that current agents struggle to jointly satisfy interdependent requirements across code implementation and actual play. Requirement-specific feedback improves GDD Fidelity by 10.9% relative to self-revision after two rounds. A2Z GameSpec-Bench assesses end-to-end specification-following ability beyond implementation judgments and provides targeted feedback to support more faithful game development. Code and datasets are available at https://a2z-gamespec-bench.github.io.