Beyond One-shot: AI Agents for Learning in Field Experiments
2026-06-01 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors studied how AI can help improve healthcare messages by learning from past experiments. They compared messages co-created by experts with AI to messages fully created by AI using data from previous tests. The AI-generated messages performed better, showing that using specific experimental data is more helpful than relying on general knowledge. They also found that common behavior theories don't always fit specific healthcare situations, so AI helps tailor messages more effectively. Overall, their work suggests AI can make experiments smarter and more useful over time.
A/B testingagentic AIbehavioral experimentationclick-through rate (CTR)Data-Information-Knowledge-Wisdom (DIKW) modellarge language models (LLMs)healthcare messagingintervention designfield experimentsbehavioral theories
Authors
Junjie Luo, Ritu Agarwal, Gordon Gao
Abstract
Organizations routinely run experiments for A/B testing, yet the data generated from one experiment is underutilized to inform subsequent intervention design. Significant barriers exist to extracting actionable knowledge from prior experimental data to inform new interventions. We study whether tool-augmented agentic AI can automatically learn from experimental data to generate new interventions in subsequent experiments. Through two-stage field experiments in healthcare prescription messaging (693,139 patient visits), we compare a Human + Chatbot method (Stage 1: behavioral experts with conversational AI co-designing 13 message variants, 444,691 patient visits) against a Tool-Augmented Agentic AI method (Stage 2: AI autonomously extracting principles from Stage 1 data to generate 17 new variants, 248,448 patient visits). The Agentic AI method, equipped with analytical tools, structured Data-Information-Knowledge-Wisdom (DIKW) reasoning agents, and transparent evidence chains, produces superior interventions: the best AI-generated message achieved a 69.8% CTR (+6.5 percentage points over baseline). Critically, our results suggest that the value comes from domain-specific experimental data, not from general reasoning ability: frontier LLMs operating without experimental data failed to predict which interventions would succeed. The field experiments also revealed that general-purpose behavioral theories used for intervention design do not extend uniformly to specific healthcare contexts, motivating an agentic AI approach to theory audits at field-experiment scale. Our research shows that tool-augmented AI can learn from experimental data and generate improved domain-relevant interventions, transforming behavioral experimentation from one-shot evaluation into a scalable system for cumulative design learning.