NaiAD: Initiate Data-Driven Research for LLM Advertising
2026-05-11 • Machine Learning
Machine LearningArtificial IntelligenceComputers and Society
AI summaryⓘ
The authors created NaiAD, a large dataset designed to help improve how large language models (LLMs) include ads in their responses without hurting user experience. They carefully built nearly 59,000 examples that combine user questions with ad-embedded answers, and developed ways to measure how well the ads work for both users and advertisers. They also introduced new techniques to generate diverse example responses and align automated scoring with human judgment. Their findings show that successful ad integration follows certain reasoning patterns that models can learn to balance both user satisfaction and commercial goals. This work provides a useful resource for building better ad-supported LLM systems.
Large Language ModelsAdvertisingDatasetUser ExperienceEvaluation MetricsIn-context LearningAutomated ScoringSemantic StrategiesData-centric Approach
Authors
Yihang Zhang, Zimeng Huang, Ren Zhai, Yipeng Kang, Tonghan Wang
Abstract
Reconciling platform revenue with user experience in LLM advertising motivates a data-centric foundation. We introduce NaiAD, the first comprehensive dataset for LLM-native advertising comprising 58,999 carefully constructed ad-embedded responses paired with user queries. NaiAD is organized around theoretically grounded evaluation metrics that separately and comprehensively capture user and commercial utility. To mitigate the dimensional collinearity of aligned LLMs, we propose a decoupled generation pipeline that produces structurally diverse samples, ranging from responses that explicitly disentangle stakeholder utilities to responses that are uniformly strong or weak across dimensions. We further provide score labels calibrated by a Variance-Calibrated Prediction-Powered Inference (VC-PPI) framework, aligning automated scoring with human annotations. Mechanistic analyses reveal that successful ad integration relies on reasoning paths that cluster into four distinct semantic strategies. Models leveraging NaiAD internalize these strategies to simultaneously improve user and commercial utility, while enabling independent control over these distinct objectives via in-context learning. Together, these results position NaiAD as a foundational infrastructure for developing future LLM-native ad systems.