New method improves evaluation of personalized ai responses
Neither Over- nor Under-Constrained: Evaluating LLM Personalization under a Constraint Qualification Audit
Artificial Intelligence
Summary
Evaluating how well large language models (LLMs) tailor their responses to individual users is tricky because current tests either limit responses too much or too little, causing mistakes in judging quality. The authors present ConQual-Audit, a three-step method that extracts, cleans up, and checks the rules that a good personalized response should follow. This method performs better than previous ones, producing fewer errors while using fewer but better rules. Their approach can also improve existing evaluation tools by refining the rules they use.
What this means in practice
- •For ai developers: Improve testing of personalized AI responses to reduce false error rates during model evaluation.
- •For language model maintainers: Refine constraint sets in model evaluation tools to better assess response quality and personalization accuracy.
Authors
Xuran Li, Guanqin Zhang, Imran Razzak, Hakim Hacid, Eleanna Kafeza, Hao Xue, Flora D. Salim
Abstract
Accurate evaluation of LLM personalization is hindered by current evaluation paradigms that over- or under-constrain the response space, yielding frequent false rejections and false acceptances. To reduce these false rates, we introduce ConQual-Audit, an audit-based framework that evaluates LLM personalization through a three-stage process: (1) extracting semantic constraints from user profiles and task-grounded reference; (2) qualifying constraints by pruning unnecessary constraints and recovering missing constraints; and (3) verifying candidate responses against the audited constraints via natural language inference. Across PrefEval and LaMP, ConQual-Audit outperforms nine baselines, attaining top AUCs (0.88 and 0.90) and the lowest false rate (0.21) on the leanest constraint sets at an amortizable audit cost. Crucially, this constraint audit is portable: integrating it into ExPerT and PREF helps revise smaller, higher-quality constraint sets to reduce their evaluation false rates.