LLMs show changing social bias in ongoing user conversations
DyMT-ESB: Dynamic Multi-Turn Evaluation of Social Bias in User-LLM Interactions
Computation and Language
Summary
Large language models (LLMs) can show social biases when talking with people. This study looks at how these biases can change during multi-turn conversations where the user's next question depends on the model's previous answer. The authors found that biases can appear later, fade, then come back, rather than just getting worse over time. This means evaluating bias in these models needs to consider how conversations evolve turn by turn.
What this means in practice
- •For chatbot developers: Design evaluation tools that detect social bias appearing at any turn during ongoing user conversations with language models.
- •For content moderation teams: Monitor conversational AI interactions to identify late-emerging and reappearing biases for safety and compliance purposes.
Authors
Rem Hida, Masahiro Kaneko, Daisuke Oba, Danushka Bollegala, Naoaki Okazaki
Abstract
Warning: This paper contains examples of stereotypes and social bias. LLMs are increasingly used in interactive settings by the general public, making the evaluation of model behavior in multi-turn conversational scenarios important for safety, including stereotyping-related harms. However, existing multi-turn social bias evaluations often rely on pre-specified or template-based user inputs that do not adapt to model responses and typically assume a fixed dialogue length in advance. In this paper, we study social bias dynamics in response-conditioned multi-turn interactions using a controlled evaluation protocol that generates follow-up user queries from the evolving dialogue history and allows evaluation over variable numbers of turns. Experimental results show that LLMs exhibit social bias even in coherent, response-conditioned multi-turn interactions, revealing late-emerging bias, non-monotonic bias patterns, and bias re-emergence. These results motivate evaluations that extend beyond fixed-turn, pre-scripted protocols. Our findings highlight the importance of analyzing social bias as a turn-level dynamic phenomenon.