Unipolicy balances multiple goals for better search ads
UniPolicy: Unified Objective-Specific Policies for Generative Search Advertising
Computation and Language
Summary
Search ads must meet many goals at once, like showing relevant ads, encouraging clicks, and making money. The authors propose UniPolicy, a new method that separates and manages these goals within one system to avoid conflicts that hurt results. Their approach lets the system pick ads based on different objectives and combines feedback to improve choices. Tests show UniPolicy works better than previous methods, boosting click rates, revenue, and user experience without slowing down service.
What this means in practice
- •For advertising platform engineers: Implement multi-objective policy alignment to improve ad relevance, click performance, and revenue simultaneously without latency issues.
- •For recommendation system designers: Adapt hierarchical multi-policy frameworks to balance varied user and business objectives in content ranking and selection.
Authors
Kun Yao, Yuhang Zhou, Yichi Zhang, Zeliang Tong, Shengri Xue, Haitao Wang, Siyu Lu, Qianlong Xie, Xingxing Wang
Abstract
Search advertising connects user intent with commercial content and plays a critical role in platform monetization. Recent systems typically align pretrained generative models with a single business reward, such as eCPM, or use naive reward fusion for preliminary multi-objective alignment. However, an ideal search advertising system must jointly account for heterogeneous objectives, including relevance, click propensity, and commercial value, to balance user experience and business value while mitigating globally suboptimal performance caused by gradient competition. We propose UniPolicy, an objective-aware multi-policy alignment framework. UniPolicy combines objective-specific prefix tokens, sparse MoE-LoRA routing, and objective-specific residual FFNs to hierarchically decouple parameters within a shared backbone, providing differentiated parameter and policy-expression spaces for different business objectives. It further constructs pairwise preferences from multi-stage behavioral feedback, supplementing the relative preference information in exposed-but-unclicked samples and strengthening the relative advantage of clicked candidates in the generation distribution. At inference, UniPolicy supports parallel, business-customizable multi-policy beam search, flexibly allocating candidate quotas across objectives under a fixed retrieval budget. Large-scale offline experiments show that UniPolicy delivers balanced improvements across multiple metrics while preserving retrieval quality, outperforming single-objective reinforcement learning and naive reward-fusion baselines. In a 7-day online A/B test on a real search advertising system, UniPolicy improves CTR by 0.71%, RPS by 1.58%, and advertising revenue by 1.32%, while maintaining stable serving latency.