LLM agents tested across multiple eurogames reveal varied strengths
A Unified Arena for LLM Agents across Diverse Eurogame
Artificial Intelligence
Summary
It can be hard to know how well AI agents make decisions in different competitive games. The authors created DECE, a platform that tests AI agents in ten different board games with similar rules and goals. They ran over a thousand matches with various AI models to see which ones played best overall and in specific games. They found some agents do well across many games, but performance can change depending on how the AI is set up and how much computing resources it uses. This work helps compare AI players more fairly by looking at both how well they play and how efficient they are.
What this means in practice
- •For game developers: Benchmark AI opponents in multiple strategic games using a common evaluation platform to find the best-performing agent configurations.
- •For ai system designers: Compare the tradeoffs between performance and resource use of LLM-based agents in competitive multi-agent scenarios.
Authors
Fengrui Zhang, Xiang Li, Ruomeng Ding, Jiayi Yuan, Yuchen Huang, Zhaoyang Han, Xiaoliang Wang, Daoyuan Chen, Yaliang Li
Abstract
Interactive environments offer a practical way to evaluate large language model (LLM) agents, and competitive games extend this evaluation to decisions shaped by competing goals and other agents' choices. Yet existing evaluations confined to individual games or execution frameworks leave unclear which competitive advantages persist across settings. We introduce DECE (Diverse Eurogame Competitive Environments), a unified arena spanning ten Eurogames with shared interfaces and evaluation protocols. These games combine lasting strategic commitments, competition over shared resources, and multiple routes to victory. Our primary study evaluates 12 model-harness configurations, spanning nine models and four execution harnesses, in 480 three-player matches across ten games. The main and supplementary studies comprise 1,032 completed matches in total. These comparisons reveal broad cross-game ranking agreement alongside game-specific strengths. Matched studies show that the same model can perform differently across execution harnesses, adding another source of variation to these profiles. These performance advantages do not consistently translate into better token or cost efficiency, making resource use an additional consideration when comparing configurations. Together, the findings support evaluating complete agent configurations by both their competitive performance across games and the resources needed to achieve it.