Neuroevolution improves game test agents using scenario models
Scenario-Driven Neuroevolution: Using Models to Guide Test Generation for Games
Software EngineeringNeural and Evolutionary Computing
Summary
Testing video games automatically is hard because the tests need to understand the game well and handle random changes. The authors improved an existing method, Neatest, by letting testers create simple models that describe how the game should be played. Instead of trying to cover every part of the game’s code, the new approach trains test agents to follow these desired behaviors. This method helped create better test agents for 13 different games and also covered more of the code than before.
What this means in practice
- •For game developers: Generate intelligent agents that test games by following scenarios defined by game designers, improving testing quality and coverage.
- •For software testing teams: Create robust automated tests for complex interactive software by combining program analysis and model-based scenario guidance.
Authors
Gijs van Cuyck, Patric Feldmeier, Jan Tretmans, Gordon Fraser
Abstract
Automatically generating test inputs for games is challenging, as test generators must master the game to reach advanced program states while also ensuring robustness against the heavy program randomisation inherent to games. The test generator Neatest therefore optimises test suites consisting of neural networks that reach advanced program states and are robust to program randomisation, as they generate test inputs dynamically based on the current program state. Neatest is a white-box testing approach that aims to generate a network agent for each yet-uncovered statement or branch of the code using neuroevolution. Due to this iterative test generation approach, the algorithm does not scale well to larger programs that may contain thousands of branches. Furthermore, covering every statement or branch in a game often does not correspond to playing the game as intended. To alleviate these shortcomings, we propose combining Neatest with a model-based testing approach that allows game testers to define test scenarios via abstract game models. The test generator then no longer optimises networks to reach all branches or statements of a program, but instead trains networks to replicate the concrete desired testing behaviour expressed by the abstract game model. An evaluation on 13 Scratch games across varying genres demonstrates that Neatest, combined with model-based testing, is able to optimise agents that replicate the desired gameplay behaviour defined in the game models while increasing achieved branch coverage by 7% compared to the traditional code-guided Neatest approach.