Summary
Playing video games well requires a mix of skills like understanding visuals, following instructions, planning goals, and precise actions over different time lengths. Existing AI game datasets don't cover many games or lack clear instructions, making it hard to measure progress broadly. The authors created GameHorizon, a collection of tools, data, and tests that capture gameplay skills at short and long timescales with aligned videos and instructions from expert players across 21 games. They tested many AI models and found big differences in how well they handle tasks of varying difficulty, offering a clear way to compare AI game-playing abilities. This work provides a common yardstick to help future AI research improve game-playing agents.
What this means in practice
- •For video game developers: Evaluate how well AI agents play different games over short and long tasks using a large-scale standardized dataset and tests.
- •For robotics engineers: Assess AI planning and action control capabilities over multiple time scales that can transfer to robotics tasks requiring complex planning.
Authors
Yiran Wang, Xingyilang Yin, Junfu Pu, Guangzhi Wang, Kaifeng Li, Mingyu Ouyang, Huiqiang Sun, Lingen Li, Cheng Cheng, Wangbo Yu, Honghao Chen, Xiaodong Cun, Chi-Man Pun, Zhiguo Cao, Ying Shan
Abstract
Modern video games provide a measurable testbed for AI models, combining abilities of visual understanding, instruction decomposition, goal planning, and precise action control over multiple temporal horizons. Existing datasets and benchmarks, however, either cover a narrow range of games, lack language instructions, or rely on high-variance online rollouts. To address these challenges, we introduce GameHorizon, a unified data and evaluation suite that measures gameplay capabilities at different horizons for diverse model families. GameHorizon Suite consists of three components. First, GameHorizon-Annotator is a scalable and automated annotation pipeline for multi-horizon instructions. Second, utilizing the pipeline, we construct GameHorizon-Data, the first large-scale AAA gameplay dataset with temporally aligned videos, player actions, and multi-horizon instructions. It comprises 5,000 hours of recordings from 21 games, collected by 100 human expert players. Third, we build GameHorizon-Bench with reproducible offline and stepwise online testing. The offline track enables reproducible evaluation using thousands of standardized questions organized into three primary tasks and a series of diagnostic variants, while the online track tests whether offline scores reflect actual gameplay capabilities and localizes failures to specific steps within long-horizon gameplay. Based on our GameHorizon Suite, we evaluate 47 models through more than one million model invocations, revealing a meaningful hierarchy of task difficulty and pronounced differences in model capabilities. Our work can provide a standardized yardstick for evaluating gameplay capabilities across horizons and model families. We will release our dataset, annotator, and benchmark to facilitate future research.