Go language model struggles to link terms with gameplay moves
Recognizing Atari and Assessing Sente: Controlled Evaluation Tasks for Language-Based Go Agents
Computation and Language
Summary
The paper looks at how well a language model trained to understand the game Go can recognize key game terms and relate them to moves in the game. The authors focus on two important concepts: 'atari,' a state threatening capture of stones, and 'sente,' a situation giving the player the initiative. They find that even after training, the model has difficulty spotting changes in 'atari' and evaluating moves based on strategy. The model’s predictions for terms sometimes influence its move choices even if these choices don’t match the best moves according to reference evaluations.
What this means in practice
- •For ai game developers: Improve game AI by testing if language-based models correctly understand game concepts when making move decisions in Go.
- •For natural language processing engineers: Use the controlled evaluation framework to assess how well domain-specific language models connect terminology to precise contextual meanings beyond Go.
Authors
Taiyi Chen, Iskar Deng, Amandalynne Paullada
Abstract
Domain terminology provides concise descriptions of game states and strategic relationships, but whether game-playing language models can accurately recognize these concepts and relate them to move evaluation remains unclear. We construct a controlled evaluation set centered on atari and its relationship to sente in Go, examining terminology recognition under local board changes and move evaluation under different strategic conditions. We conduct our tests on LoGos, a Go language model, after fine-tuning it with terminology and commentary supervision. The model struggles to recognize changes in the atari relationship and assess its strategic implications. Its terminology predictions are associated with move preferences even when those preferences contradict the reference evaluation. Our controlled tests provide a framework for examining whether language-based Go agents connect domain terminology with precise board relationships and strategic move values.