Unified benchmark reveals ai strengths and weaknesses in game commentary
GameCommBench: A Unified Benchmark and Type-Aware Evaluation for AI-Generated Game Commentary
Artificial IntelligenceComputation and Language
Summary
AI systems that generate commentary for games need to understand what is happening, why it matters, and how to explain it. The authors created GameCommBench, a single benchmark that covers many types of games and commentary styles, helping to compare AI performance fairly. They also designed a method called TACE to evaluate different commentary types clearly. Testing current AI systems showed they struggle most with live observations and strategic insights. These tools help identify where AI can improve in providing game commentary.
What this means in practice
- •For game development studios: Evaluate and improve AI commentators across various game types using a unified benchmark and detailed feedback on commentary types.
- •For sports analytics companies: Assess AI tools that generate live sports commentary to pinpoint strengths and weaknesses in observation and strategic analysis.
Authors
Qirui Zheng, Zhengteng Lin, Yunyi Xiao, Junhao Li, Keyuan Cheng, Xingbo Wang, Yongyi Wang, Lingfeng Li, Yunlong Lu, Wenxin Li
Abstract
Game commentary is an open-ended generation task requiring multimodal perception, strategic reasoning, and contextual knowledge. Existing AI-Generated Game Commentary (AI-GGC) studies remain fragmented across games, modalities, and evaluation protocols, while overlap-based or holistic evaluators fail to capture the functional heterogeneity of commentary. We introduce \textsc{GameCommBench}, a unified benchmark spanning board games, sports, and esports, with commentary aligned to heterogeneous game contexts and annotated by commentary type. We further propose Type-Aware Commentary Evaluation (TACE), a structured framework for evaluating different types of commentary. We then validate TACE for reliability and human agreement, and use it to benchmark representative AI commentators. Results reveal non-uniform capability profiles, with live observation and strategic analysis emerging as major bottlenecks. Together, \textsc{GameCommBench} and TACE provide a diagnostic foundation for comparable and interpretable AI-GGC evaluation.