Multi agent system improves natural language recommendation explanation evaluation
MAREval: A Multi-Agent Framework for Evaluating Natural Language Recommendation Explanations
Artificial IntelligenceInformation Retrieval
Summary
Evaluating explanations given by recommendation apps is important but usually requires human reviewers, which is slow and costly. The authors created MAREval, a system using several AI agents that talk and coordinate to better judge explanation quality, making the evaluation more reliable and consistent. They tested MAREval with public and private shopping data, finding it matches human judgment better and is more stable. This approach can scale to help recommendation platforms improve user trust and engagement.
What this means in practice
- •For recommender system engineers: Enhance evaluation of recommendation explanations by integrating multi-agent large language model orchestration for more consistent and human-aligned quality assessment.
- •For online retail platform operators: Improve user engagement and revenue by adopting robust automated evaluation of natural language explanations in recommendation features at scale.$Commercial implications: Enables scalable evaluation services that increase retail sales through better user trust and personalized recommendations.
Authors
Reza Yousefi Maragheh, Jayesh Uddhav Kudase, Aysenur Inan, Ramin Giahi, Kai Zhao, Parth Parikh, Jianpeng Xu, Jason Cho, Evren Korpeoglu, Sushant Kumar
Abstract
Evaluating natural language explanations in recommender systems is essential for fostering user trust, transparency, and engagement. However, existing evaluation approaches like human evaluations, while accurate, are resource-intensive and impractical at the scale required by modern recommendation platforms. Also, automated methods using single-agent LLMs suffer from prompt sensitivity and inconsistent outputs. To address these challenges, we propose MAREval, a structured multi-agent framework for evaluating recommendation explanations using large language models. MAREval orchestrates (i) a planner agent that uses a novel Chain of Debate (CoD) prompting strategy to coordinate agent roles and enforce logically consistent evaluation plans; (ii) a moderator agent that regulates discussions by mitigating prompt drift; and (iii) an arbitrator agent that aggregates outputs from multiple evaluation rounds and (iv) a Monte Carlo sampling method, improving robustness and alignment with human judgment. We conduct comprehensive evaluations on both public (TopicalChat) and proprietary recommendation datasets, demonstrating that MAREval outperforms state-of-the-art baselines. Comprehensive experiments on a public benchmark and a proprietary e-commerce dataset show that MAREval improves alignment with human judgments over strong singleand multi-agent baselines. Stability analyses indicate substantially lower variability across repeated trials. In a large human-annotation gate, MAREval meets production quality thresholds where prior evaluators fall short, and online A/B testing demonstrates statistically significant improvements in engagement and revenue metrics. These results establish MAREval as a scalable and reliable solution for human-aligned evaluation of recommendation explanations in real-world systems