Improving online ad tests by filtering biased training data
BAFF: Bid-Aware Filter Family for Mitigating Training Data Interference in RTB A/B Tests
Machine Learning
Summary
When companies test new online advertising models, they often use the same data that has been influenced by existing models, which can lead to biased results. The authors propose a way to filter this data based on how much competing models differ in their ad choices and bids, helping to reduce this bias without losing too much data. They show that their method better maintains important business metrics compared to existing approaches. Their work highlights that the best solution depends on the specific testing setup.
real-time biddingA/B testingtraining data biasad rankingbid pricinglog splittinglog sharingdemand-side platformclick-through ratecost per click
Authors
Jeonglyul Oh, Ikkyu Choi, Inseop Youn, Youngjae Kim
Abstract
In online A/B tests for real-time bidding (RTB), control and treatment models are typically trained on a shared serving log that includes data generated by the counterpart model. This shared-log training biases each model's training data through two channels: the counterpart model may have selected a different ad from the ad-candidate pool (ad-ranking disagreement) and may have bid a different price (bid-pricing disagreement), potentially distorting the A/B test outcome. Log-splitting eliminates the bias but sacrifices training data; log-sharing retains all data but leaves the bias unaddressed. We formalize the Bid-Aware Filter Family (BAFF), a class of (k,l)-parameterized hard filters that controls tolerance to each channel independently, providing a structured search space between these two extremes. We further propose a three-stage online measurement protocol that enables evaluating data-sharing strategies by their deviation from an interference-free reference model in production. In offline simulation, a (k,l) sweep surfaces operating points with smaller deviation from the interference-free reference model than both log-sharing and log-splitting. In a live RTB deployment on a demand-side platform (DSP), filter-based variants preserve the reference model's business metrics (e.g., CPC, CTR) more closely than both baselines. The best operating point is setting-dependent, underscoring the practical value of the search space itself.