Benchmarking large multimodal model agents in real-time multiplayer shooter games
ROAR: Benchmarking LMM Agents under Real-Time Constraints in Multiplayer Action-Shooter Games
Multiagent Systems
Summary
Many computer programs called large multimodal models (LMMs) can think and act in video game worlds, but most tests pause the game while the program decides what to do next. This isn’t like real-time games where the world keeps moving and decisions must be quick. The authors made a new multiplayer shooter game setup called ROAR that tests how well these programs handle fast-paced decisions without stopping the game or each other. They also created a special interface letting the game run continuously while each program thinks independently. This setup helps build and test smarter game-playing programs that work in real-time with other players.
What this means in practice
- •For game developers: Test and improve AI agents that must make fast decisions alongside or against human players in real-time multiplayer games.
- •For robotics developers: Develop AI agents for robots that operate in dynamic environments requiring asynchronous perception and action without pausing the world.
Authors
Dawei Wang, Di Zhao, Jianran Yu, Chengming Zhou, Marci Chi Ma, Xiaoyang Liu, Beining Zhang, Shuliang Chu, Yun Sing Koh, Gary Ushaw, Richard Davison
Abstract
Large multimodal models (LMMs) are increasingly deployed as agents that perceive, reason, and act in interactive multi-agent environments. Yet most existing benchmarks for LMM-based agents in multi-agent settings remain turn-based: the environment and other agents pause while an agent performs inference. Such protocols obscure a fundamental challenge of real-time constraints, where the world continues to evolve and delayed decisions may become outdated before execution. Motivated by this limitation, we introduce ROAR, a multiplayer action-shooter game environment for evaluating LMM-based agents in multi-agent settings under real-time constraints. ROAR covers cooperative, competitive, mixed cooperative-competitive, and basic single-agent scenarios. We further propose a Multi-Agent Asynchronous Gym interface that decouples environment evolution from agent reasoning by running the environment and each agent in separate threads, enabling the environment to advance at fixed time intervals without waiting for LMM inference and allowing agents to reason independently without blocking one another. ROAR also features convenient deployment, multimodal observations, and customizable levels. ROAR offers a practical testbed for advancing LMM-based agents beyond static, turn-based evaluation toward real-time multi-agent decision-making.