Large language models struggle to weigh trust and accusations in Werewolf games
Do LLMs Trust the Accuser or the Accusation? Measuring Belief Shifts in Werewolf
Artificial IntelligenceComputation and Language
Summary
Some language models try to play the game Werewolf by listening to players' accusations and deciding who might be a bad player. The paper finds that bigger models are better at spotting the real bad players using game history, but they still get strongly influenced by who is accusing and who is accused. These models tend to trust the accuser more if they are usually trusted, even if that accuser is actually a bad player, and they become more suspicious of the accused. Overall, the models find it hard to balance what is said and who says it in tricky conversations. The researchers made a tool to measure this behavior for many different models.
What this means in practice
- •For game ai developers: Improve AI agents playing social deduction games by analyzing how they update beliefs based on accusations and trust.
- •For conversational ai designers: Design dialogue systems that better weigh source trust and message content in multi-party strategic conversations.
Authors
Yu-Yu Yang, Ti-Rong Wu, Hung Guei, Hsing-Yu Chen, I-Chen Wu
Abstract
Social-deduction games such as Werewolf are increasingly used to evaluate LLM agents, but existing evaluations often rely on final game outcomes. We propose a belief-shift evaluation benchmark in Werewolf for analyzing communication skills through belief updating. Using LLM-played games, we annotate suspicion and accusation messages and measure how an observing village-side model's beliefs change after each message. We evaluate 40 open-weight LLM configurations on 1,224 annotated messages. Our results show that larger models better distinguish true wolves from villagers based on game history, but accusations still strongly influence their beliefs. Models become more suspicious of the accused target and less suspicious of the accuser, especially when the accuser is trusted, even if the accuser is wolf-aligned. Larger models better resist accusations from accusers they already distrust. Overall, our findings suggest that current open-weight LLMs up to 120B parameters still struggle to integrate accusation content with source trust in strategic communication. Our benchmark and code are available at https://rlg.iis.sinica.edu.tw/papers/werewolf-accusation-benchmark.