E commerce agents improve help with better reward models
E-GenRM: Generative Reward Modeling for Open-Ended E-Commerce Shopping Agents
Information Retrieval
Summary
Online shopping assistants are expected to handle more open-ended and complex tasks, but current systems struggle because they learn from limited examples and simple reward systems. The authors introduce E-GenRM, a method that uses real user behavior and conversations to better understand and judge shopping preferences over longer periods. They train the model with both supervised learning and a special reinforcement method that rewards accuracy in matching user preferences and facts. Their method improves performance on tests and increases user satisfaction and shopping help on a large shopping platform.
What this means in practice
- •For e-commerce platform teams: Increase shopping assistant effectiveness and user satisfaction by integrating E-GenRM for better reward learning from real user interactions.$Commercial implications: Allows e-commerce platforms to deploy advanced conversational agents that better understand shopper preferences, improving user experience and conversion rates.
- •For mobile app developers: Enhance long-horizon conversational AI in shopping apps with generative reward models that evaluate preferences and factual claims accurately.
Authors
Ying Sun, Qiuyu Zhao, Xiao Wang, Xiyu Zhao, Gege Shi, Zenghui Sun, Jinsong Lan, Haihong Tang, Bo Zheng
Abstract
We present E-GenRM (E-commerce Generative Reward Model), a generative reward model tailored to e-commerce shopping agents. User expectations are shifting from simple product queries to open-ended shopping tasks. However, existing trajectories and reward designs remain largely bounded by current agent usage patterns and known failures, providing limited guidance for further capability improvement. These limitations call for an integrated reward modeling framework for open-ended shopping agent tasks. On the data side, E-GenRM leverages naturally occurring user behavior across the shopping app to construct diverse, long-horizon conversational trajectories beyond established usage patterns. On the modeling side, E-GenRM first learns from behavior-grounded evaluation chains through supervised fine-tuning. It is then optimized through criterion-guided reinforcement learning, which rewards correct preference judgments and evidence-supported factual claims under each criterion. Personalized hard-negative training further improves its sensitivity to user preferences reflected in shopping behavior, beyond explicit requirements. We additionally develop an active evidence retrieval extension that acquires missing e-commerce information for evaluation. On E-GenRM-Bench, E-GenRM-27B achieves 87.40\% preference accuracy, outperforming the strongest baseline by 6.97 percentage points. In a 14-day online A/B test on Taobao, a shopping agent post-trained with E-GenRM improves the effective shopping-assistance rate by 1.39 percentage points and user satisfaction by 2.33 percentage points.