Can AI Agents Simulate A/B Test Outcomes? A Validation Framework for Agentic Experimentation
2026-08-03 • Computation and Language
Computation and LanguageArtificial Intelligence
AI summaryⓘ
The authors explore whether AI models can simulate A/B test results before running costly real experiments. They create a framework called Simulated Randomized Controlled Trial (S-RCT) to measure errors from AI predictions and sampling. Testing on 67 past marketing experiments, they find that basic AI models predict the direction of effects well but tend to overestimate their size. They improve accuracy by calibrating with pre-test data and using methods that expose the AI to both test conditions. The work shows potential for AI to reduce the time and resources needed to evaluate new features but also discusses current limitations.
A/B testingRandomized Controlled TrialSimulated RCTAI agentbehavioral profileseffect sizecalibrationpre-period datawithin-subject designmeasurement error
Authors
Stefan Hut, Lorenzo Masoero
Abstract
A/B testing remains the standard for rolling out new features in the technology industry. Each experiment, however, consumes real traffic, engineering effort, and weeks of wall-clock time. Can AI agents---conditioned on behavioral profiles and contextual descriptions of the intervention---simulate outcomes accurately enough to vet candidate treatments before committing live traffic? We formalize this question as a \emph{Simulated Randomized Controlled Trial} (S-RCT) and derive a two-layer error decomposition that separates agent approximation error from subsampling error, enabling targeted improvements to each. The framework is agent-agnostic: any behavioral model---from a fine-tuned specialist to a general-purpose foundation model---can serve as the simulation engine. Validated on 67 historical marketing A/B tests, a baseline S-RCT using an off-the-shelf foundation model captures directional signal (sign overlap 0.70) but systematically overshoots effect magnitudes. A two-phase pre-period calibration protocol reduces the squared prediction error (after removing irreducible measurement noise) by ${\sim}77\times$; a within-subject design---where each agent is exposed to both arms---reduces standard errors by ${\sim}2.4\times$. We discuss limitations of the current approach and identify applications where experimenters stand to benefit from agentic signals.