LLM agents tested on video game speedrunning challenge benchmark
SpeedrunBench: Challenging LLM Agents with Video Game Speedrunning
Artificial Intelligence
Summary
The paper looks at how well large language model (LLM) agents can come up with clever strategies by testing them on video game speedrunning, a challenge where players find the fastest ways to finish games. The researchers created a new benchmark called SPEEDRUNBENCH that involves 9 different games requiring agents to improve their performance repeatedly and think ahead. Results show that while these agents can match human experts in simple games, they still struggle to compete with humans in more complex, longer games. This benchmark helps measure how well agents can form strategies and keep finding better solutions.
What this means in practice
- •For game ai developers: Evaluate and improve AI agents’ long-term strategic planning using SPEEDRUNBENCH’s diverse video game speedrunning tasks.
- •For robotics software teams: Test and refine robots’ sequence planning and adaptive strategy abilities through the benchmark’s demand for repeated improvement and reflection.
Authors
Yoshinari Fujinuma, Keisuke Kamahori, Ryuto Koike, Abdelrahman Madkour, Varun Prashant Gangal, Monty Bichouna, Martyna Markiewicz, Shivani Jain, Duncan Curtis, Rebecca Qian, Anand Kannappan
Abstract
Frontier LLM agents have been shown to be capable of solving increasingly complex tasks for which humans have measurable solutions. This begs the pertinent question of whether LLM agents can go beyond what humans have already solved. The ability to develop sophisticated strategies to tackle consequential problems becomes paramount as well-trodden, human-developed solutions become insufficient for problems for which we lack context or enough training data. We study agents' capability of such strategy formation through the communal practice of video game speedrunning. In speedrunning, practitioners compete to find the fastest way to complete a video game under certain conditions, and in so doing uncovering interesting unorthodox play styles that require a thorough understanding and mastery of the underlying game mechanics. We introduce SPEEDRUNBENCH, a benchmark that evaluates frontier LLM agents across 9 different games. To perform well in this benchmark, agents must repeatedly improve their strategy, reflect on their performance, exploit their gained knowledge, and reason across a long-horizon of actions to improve on an increasingly difficult problem: being faster than themselves and everyone else. Our experiments show that while frontier agents approach human world records in simple platformer games, they remain behind human performance on longer, more complex games under practical budgets. These results suggest that SPEEDRUNBENCH is a useful testbed for studying agents' strategy formation capabilities as well as being a saturation-resistant evaluation measure, as there is almost always a faster completion time waiting to be discovered.