Papers for

conversational ai designers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Large language models struggle to weigh trust and accusations in Werewolf games

Do LLMs Trust the Accuser or the Accusation? Measuring Belief Shifts in Werewolf

Abstract: Social-deduction games such as Werewolf are increasingly used to evaluate LLM agents, but existing evaluations often rely on final game outcomes. We propose a belief-shift evaluation benchmark in Werewolf for analyzing communication skills through belief updating. Using LLM-played games, we annotate suspicion and accusation messages and measure how an observing village-side model's beliefs change after each message. We evaluate 40 open-weight LLM configurations on 1,224 annotated messages. Our results show that larger models better distinguish true wolves from villagers based on game history, but accusations still strongly influence their beliefs. Models become more suspicious of the accused target and less suspicious of the accuser, especially when the accuser is trusted, even if the accuser is wolf-aligned. Larger models better resist accusations from accusers they already distrust. Overall, our findings suggest that current open-weight LLMs up to 120B parameters still struggle to integrate accusation content with source trust in strategic communication. Our benchmark and code are available at https://rlg.iis.sinica.edu.tw/papers/werewolf-accusation-benchmark.

Fri 11 SeptArtificial IntelligenceComputation and Language
The gist
Some language models try to play the game Werewolf by listening to players' accusations and deciding who might be a bad player. The paper finds that bigger models are better at spotting the real bad players using game history, but they still get strongly influenced by who is accusing and who is accused. These models tend to trust the accuser more if they are usually trusted, even if that accuser is actually a bad player, and they become more suspicious of the accused. Overall, the models find it hard to balance what is said and who says it in tricky conversations. The researchers made a tool to measure this behavior for many different models.
Open 2609.12446v1