One Model, Multiple Goals: Adaptive Multi-Objective Learning for E-commerce Dialogue Systems
2026-06-08 • Computation and Language
Computation and Language
AI summaryⓘ
The authors developed a system called MORE to help chatbots in online shopping better understand user information and respond naturally at the same time. They found that trying to optimize for both reasoning and natural language together causes unstable learning, so they treated reasoning as a guiding constraint instead of mixing rewards. Their method adjusts how much it cares about different language qualities during training and produces good responses quickly. Tested on real data, MORE improved sales conversions and user satisfaction while needing fewer human handoffs, and it performed closer to human agents in effectiveness.
dialogue systemse-commercereinforcement learningmulti-objective optimizationpolicy optimizationreward shapingresponse generationuser profilingnatural language processingMultiWOZ
Authors
Mingzhe Li, Jing Xiang, Enguo Zhou, Lang Gao, Tai Li, Qishen Zhang, Xiangliang Zhang, Xiuying Chen
Abstract
Dialogue systems in e-commerce scenarios often need to satisfy multiple objectives: accurately reasoning over user profiles (e.g., eligibility, credit limit) to ensure correct decision-making and user state interpretation, while also generating natural and faithful responses. These goals are complementary but not identical. In this work, we propose MORE, an adaptive Multi-Objective REinforcement learning framework that jointly optimizes reasoning accuracy and linguistic naturalness. Our preliminary experiments show that directly mixing rewards with diverging optimization dynamics can cause oscillations and unstable learning. Thus, instead of optimizing a single mixed reward, we treat reasoning functions as constraints that guide policy optimization. At inference time, the system directly generates responses without explicit reasoning steps, while still benefiting from reasoning-enhanced scaffold and avoiding additional inference overhead. To better balance linguistic objectives during response generation, we introduce an adaptive multi-reward mechanism that aggregates signals such as fluency and naturalness and dynamically reweighs them via gradient feedback. We evaluate MORE on two real-world dialogue systems at ByteDance and the MultiWOZ 2.2 benchmark, where it consistently outperforms strong baselines. In 14-day online experiments on ByteDance production traffic, MORE improves overall and reached conversion by 16.53% and 30.09%, while increasing user satisfaction and reducing handoff rates. Notably, in a human-machine comparison, MORE recovers about 60% of the incremental conversion lift achieved by human agents.