Bid aware filters reduce bias in real time bidding tests
BAFF: Bid-Aware Filter Family for Mitigating Training Data Interference in RTB A/B Tests
Machine Learning
Summary
In online ad auctions, companies test new bidding models by comparing them in A/B tests using shared data. However, this shared data can be biased because each model influences which ads and bids are recorded. The authors introduce a family of filters called BAFF that selectively controls how much data is shared between models to reduce this bias. Their method finds a middle ground between keeping all data and splitting logs completely, improving the accuracy of test results while preserving important information. They show in simulations and live tests that these filters keep the business outcomes closer to ideal, unbiased benchmarks.
What this means in practice
- •For advertising platform engineers: Improve the fairness and accuracy of A/B tests by applying bid-aware filters to training data shared between competing real-time bidding models.
- •For online marketing teams: Use insights from bid-aware filtering to better interpret the performance results of ad campaign experiments that use shared user logs.
Authors
Jeonglyul Oh, Ikkyu Choi, Inseop Youn, Youngjae Kim
Abstract
In online A/B tests for real-time bidding (RTB), control and treatment models are typically trained on a shared serving log that includes data generated by the counterpart model. This shared-log training biases each model's training data through two channels: the counterpart model may have selected a different ad from the ad-candidate pool (ad-ranking disagreement) and may have bid a different price (bid-pricing disagreement), potentially distorting the A/B test outcome. Log-splitting eliminates the bias but sacrifices training data; log-sharing retains all data but leaves the bias unaddressed. We formalize the Bid-Aware Filter Family (BAFF), a class of (k,l)-parameterized hard filters that controls tolerance to each channel independently, providing a structured search space between these two extremes. We further propose a three-stage online measurement protocol that enables evaluating data-sharing strategies by their deviation from an interference-free reference model in production. In offline simulation, a (k,l) sweep surfaces operating points with smaller deviation from the interference-free reference model than both log-sharing and log-splitting. In a live RTB deployment on a demand-side platform (DSP), filter-based variants preserve the reference model's business metrics (e.g., CPC, CTR) more closely than both baselines. The best operating point is setting-dependent, underscoring the practical value of the search space itself.