Human preferences guide best ad placement in large language model replies
LLMAdBench: A Human Preference Benchmark for Advertising in LLM Responses
Artificial Intelligence
Summary
Placing ads inside responses generated by large language models (LLMs) is becoming a new way to make money, but it’s unclear where ads should go to keep users happy. The authors created LLMAdBench, a dataset where people judged pairs of LLM answers that were the same except for where an ad was placed. They found that humans prefer different ad placements depending on whether ads are clearly labeled as sponsored or not. They also showed that current AI models can’t reliably decide the best placement, but training on human data helps improve their performance.
What this means in practice
- •For advertising technology teams: Design better ad placement strategies in AI-powered chat services by using human preference data to balance advertiser and user interests.
- •For conversational ai developers: Improve language models’ responses with integrated ads by fine-tuning models on human preference benchmarks for ad positioning.
Authors
Rui Ai, Yuqing Liu, Sitao Qiu, Yun Qiao, Yuhan Wang, Jessica Xiwen Wang, Yiqi Yang, Lihong Huang, Ruiyao Sun, Kaifeng Zhang, Shengze Ding, Jiaqi He, Xinman Wang, Tianhao Gao, Jimmy Qin, Jianghao Lin, Chonghuan Wang
Abstract
Inserting advertisements (ads) into consumer-facing LLM output is emerging as a new business model, but there is little shared evidence on how such ad insertion should be evaluated or how it affects user preferences. We introduce LLMAdBench, a human-preference benchmark for studying advertising in LLM-generated content. The benchmark isolates a simple but practically important decision: given a user conversation, an LLM response, and a matched advertisement, where should the ad be placed? Our dataset compares pairs of responses that differ only in ad position while holding all other conditions fixed including the user query, base answer, advertisement, and disclosure condition. Human annotators evaluate each pair based on six criteria from both advertiser's and user's perspectives. The resulting benchmark contains more than 18000 human judgments across two disclosure conditions: explicitly labeling the ad as sponsored and merging it into the response without disclosure. We use LLMAdBench to evaluate eight frontier LLMs as preference judges and find that they are not reliable substitutes for human evaluation. Even the most stable models reverse roughly one quarter of their decisions when the presentation order is swapped, agreement across models is low, and their placement preferences differ systematically from those of human annotators. Moreover, LLMAdBench contains substantial learnable signal. In particular, a Qwen3-8B model fine-tuned on the human preferences improves substantially over its base model and outperforms all zero-shot frontier judges on the held-out prediction task. Beyond model evaluation, LLMAdBench provides quantitative evidence on the advertiser-user trade-off and shows that the sponsorship disclosure systematically changes users' preference over ad placement.