MafiaScope: Non-Invasive, Time-Resolved Belief Probing for LLM Agents in Social Deduction Games
2026-07-12 • Computation and Language
Computation and LanguageMultiagent SystemsSocial and Information Networks
AI summaryⓘ
The authors created MafiaScope, a tool that uses the game Mafia to study how well AI agents understand and predict other players' thoughts and intentions (Theory of Mind). Instead of just watching the agents play, MafiaScope asks them private questions after each move to see what they really believe, then compares these answers to what actually happened. It also offers a visualizer to track how agents' beliefs change over time and lets users explore different ‘what if’ scenarios. Their tests showed that the agents were not very good at judging how confident or suspected they were. The authors provide their software and game data openly for others to use.
Theory of MindMafia gameSocial deductionLarge Language ModelAgent behaviorCalibrationCounterfactual replayMachine learning evaluationBelief tracking
Authors
Ilia Karpov
Abstract
An LLM agent's public behaviour reveals little about its social reasoning: an agent that votes correctly may be guessing, and an agent that lies well leaves no trace of what it actually believes. We present MafiaScope, an open testbed that turns the social deduction game Mafia into a measurement instrument for machine Theory of Mind. After every public utterance, every agent privately answers a configurable set of structured probe questions; the answers never re-enter the game and are scored automatically against the ground truth the engine knows. An interactive visualizer renders the belief trajectories: impersonate mode shows the game as one agent sees it, panels chart timeline-aligned accuracy and calibration, and counterfactual replay forks any recorded step. In a 32-game DeepSeek case study with 13{,}815 parsed probe answers, stated confidence is poorly calibrated, with expected calibration error 0.17, agents over-predict being suspected 1.5 times, and a 30-fork replay experiment walks the counterfactual replay workflow end to end. Engine, viewer and a corpus of 200+ cross-model games are released under an open licence; live demo: https://karpovilia.github.io/mafiascope/; screencast: https://vimeo.com/1208920221.