Impression Share Prediction: An Offline Evaluation Task for Ranking Systems
2026-08-17 • Information Retrieval
Information Retrieval
AI summaryⓘ
The authors study how to evaluate ranking models before putting them live in online tests. They focus on predicting how a model would distribute user impressions across different goals (like clicks or views), which is important because changing this distribution can affect overall usefulness. They create a method using a causal model and machine learning to predict these impression shares from existing data, without needing live traffic. Their approach improves prediction accuracy for models seen before, and with a more advanced version, they also get better results for new models soon after deployment.
offline evaluationranking modelsimpression sharecounterfactual predictionstructural causal modelRandom Foreststatistical learningA/B testingoptimization objectivesonline evaluation
Authors
Mohsen Malmir, Houssam Nassif, Danish Nasir Shaikh, Taher Rahgooy, Murat Ali Bayir
Abstract
Offline evaluation is a major gateway before online evaluation of ranking models in A/B testing. Standard offline metrics measure predictive accuracy, but are only a surrogate for downstream utility: a model can improve them while redistributing impressions across objective buckets in ways that degrade downstream utility. No offline method surfaces these impression share shifts before online evaluation. We propose \emph{impression share prediction} as an offline evaluation task: given a candidate ranking model, predict the distribution of impressions it would produce across objective buckets - impressions grouped by optimization goal (e.g., click, video view). The task is inherently counterfactual, since the candidate has never served live traffic. We propose a structural causal model of how model predictions and delivery capacity jointly determine impression allocation, and show the counterfactual effect is identified from observational data. Building on this, we develop a statistical learning framework that predicts impression shares from a candidate's early-interaction confidence signals and current system state, trained on historical data. On data from multiple ranking model families, a Random Forest reduces L1 error by 49\% over a constant baseline for models seen during training. For held-out models, evaluated by time since first appearance, the first hour is the closest analog to true online evaluation and the hardest: the Random Forest falls below the baseline because the capacity state still reflects the prior model. An encoder-conditioned architecture that simulates a 2-hour rollout over recent auction dynamics recovers $+$22\% L1 in this regime.