LLM evaluators struggle to judge correct use of user preferences
PersEvalBench: Can LLM Evaluators Judge Contextually Appropriate Use of Persona Information?
Computation and Language
Summary
Large language models (LLMs) are being used to judge whether virtual assistants use personal information correctly, but this study shows they often fail at it. The authors made a test called PersEvalBench with many examples where assistants should remember, share, or forget user info. While people easily spot when assistants don't personalize properly, LLM evaluators often confuse missing or wrong use of preferences. Highlighting relevant user history only slightly improves their judgment, suggesting it's hard for LLMs to decide how to use personal info in conversations.
What this means in practice
- •For chatbot developers: Improve evaluation of chatbots’ personalized responses by identifying where LLM evaluators misjudge user preference use.
- •For customer support platform teams: Develop better automated tools for detecting personalization errors in multi-turn customer interactions using benchmarks like PersEvalBench.
Authors
Xinyi Liu, Hooshang Nayyeri, Emine Yilmaz, Dilek Hakkani-Tur, Charith Peris, Hari Thadakamalla
Abstract
Large language model (LLM) evaluators are increasingly used to assess personalized assistants, yet whether they can reliably judge personalization has rarely been tested. We introduce PersEvalBench, a benchmark testing whether an evaluator can recognize appropriate and inappropriate uses of user history. It contains 403 multi-episode scenarios where user information should persist, transfer, or cease to apply. The benchmark covers nine controlled error subtypes across three families: under-personalization, over-personalization, and misapplied-state errors. Human audits confirm that generated errors match their intended types and that the prescribed uses of user information are reasonable. Across five LLM evaluators, recognition of controlled personalization errors remains limited even with the full dialogue history. On the human-evaluated subset, under-personalization shows the largest gap: humans recognize 94.4% of these errors, compared with 63.9% for the strongest reported evaluator. Evaluators often recognize that a response is problematic but reverse the error direction, mistaking failure to follow an applicable preference (UNDER) for use of an inapplicable preference (OVER). Easier evidence access offers little help: highlighting the relevant history turns raises error recognition only slightly, suggesting that the remaining difficulty lies in judging what role that information should play in the response.