Reinforcement learning improves ad bidding cost efficiency at pinterest
Reinforcement Learning for Controller Parameters in the Auto-Bidding System at Pinterest
Machine LearningInformation Retrieval
Summary
Online advertising needs to spend budgets wisely while keeping costs low, but these goals can change over time and vary by campaign. Pinterest uses a common control method called PID for bidding, but fixed settings don’t adapt well to changing campaigns or traffic. The authors created a system called RLBP that uses reinforcement learning to adjust PID controller settings daily for each campaign while keeping the quick PID bidding. This approach led to lower costs without hurting revenue during tests at Pinterest.
What this means in practice
- •For online advertising teams: Adjust bidding controller parameters daily to improve cost efficiency across diverse ad campaigns without retraining models.
- •For digital marketing platform engineers: Integrate reinforcement learning as a meta-controller to optimize existing PID bid adjustment systems under changing traffic conditions.
Authors
Yichu Zhou, Qinyi Chen, Fei Feng, Jacob Loh, Mehdi Ben Ayed, Lin Yang, Andrei Curelea, Jiajing Xu
Abstract
Auto-bidding systems in online advertising must balance competing objectives-maximizing budget delivery while minimizing cost per action-across heterogeneous campaigns under non-stationary market conditions. This trade-off evolves over campaign lifespans: early on, cost optimization dominates when budgets roll over; near deadlines, budget delivery becomes urgent. While PID controllers are widely used in production for their stability and interpretability, fixed tunings cannot adapt to diverse campaigns and shifting traffic patterns. We present RLBP (Reinforcement Learning for Bidding Parameters), a hybrid architecture deployed at Pinterest that uses RL as a meta-controller to select campaign-specific PID parameters daily while retaining PID for high-frequency bid adjustments. RLBP trains a multi-head Q-network using a relative-lift reward formulation normalized against the production baseline, and performs inference-time fusion that adapts the cost-delivery trade-off to campaign states without retraining. Through online A/B testing at Pinterest, RLBP achieves a statistically significant reduction in effective cost-per-click (eCPC) with no revenue degradation, and two months of post-rollout measurements against a holdout also show consistent cost reductions. These results demonstrate that meta-controller RL can successfully augment production control systems at scale.