Tool improves large language models playing the logic game Clue
Clueing up LLMs with Tool-Augmented Deductive Reasoning
Artificial Intelligence
Summary
Large language models (LLMs) struggle with complex logical reasoning that requires keeping track of information over many steps. The authors use the board game Clue as a test to see how well these models can deduce hidden information while maintaining consistency during multi-turn gameplay. They created multiple AI agents modeled on popular LLMs to play the game and introduced a special tool that turns the game history into a clear matrix of possibilities. This tool helps the AI keep better memory and reasoning constraints, leading to improved performance. Comparing agents with and without the tool shows that external aids can enhance logical reasoning in language models.
What this means in practice
- •For ai developers: Create language model agents that maintain logical consistency across complex multi-step tasks using external state-tracking tools.
- •For game ai engineers: Develop better strategic game-playing agents that explicitly track and reason over uncertain hidden information during play.
Authors
Rebecca Ansell, Autumn Toney-Wails
Abstract
Despite recent advances in large language models (LLMs), performing logically consistent deductive reasoning over extended interactions remains challenging. Tasks that require integrating evidence across multiple reasoning steps, maintaining consistency with prior inferences, and updating beliefs under new constraints can surface limitations in current models while providing a useful testbed for evaluating reasoning enhancements. In this paper, we implement a text-based, multi-agent version of the classic board game Clue as an environment to evaluate multi-step, agentic deductive reasoning. In this setting, agents must infer hidden information from a sequence of observations, maintain consistency across turns, and reason over an evolving set of logical constraints. We instantiate six LLM-based agents (GPT-4o-mini and Gemini-2.5-Flash) as players that engage in turn-based gameplay; using three agents per model family, we establish baseline performance across repeated games. We then introduce a tool-augmented approach in which a structured possibility matrix converts implicit game state from generated reasoning logs into an explicit representation of remaining possibilities. The possibility matrix encodes extended-turn memory and deductive constraints, offloading these tasks from the agent. We compare this approach against the baseline to evaluate how tool augmentation supports reasoning quality and task success for autonomous agents in a strategic reasoning environment.