ChatGPT web citation patterns vary across modes and models
Understanding ChatGPT Citations: Evidence from Production Web Search Across Model Generations
Information Retrieval
Summary
When ChatGPT answers questions using the web, it shows some websites as sources but not all it finds. The authors studied how different versions and settings of ChatGPT choose which sites to mention. They found that some versions search and cite websites differently, and that looking at websites by exact link or by bigger groups like hostnames matters. External search results only partly match ChatGPT’s citations. This means improving which sources get cited needs to consider these differences and the way queries change.
What this means in practice
- •For search engine engineers: Improve how generative search engines decide which retrieved web pages to show as citations, by incorporating query rewriting and multiple retrieval rounds.
- •For digital marketing teams: Optimize website content visibility in AI-generated answers by targeting hostname- and entity-level presence instead of only exact URLs.
Authors
Yunxuan Jiang, Shiqi Zhang, Anqi Xing, Xiangyu Chang, Xiaokui Xiao
Abstract
Generative engines such as ChatGPT Web retrieve webpages to answer user queries, but only a subset are shown as citations. Understanding this retrieval-to-citation process is important for generative engine optimization (GEO), which aims to improve content visibility in generative-engine responses. However, prior studies mainly analyze final responses, leaving the internal execution process and its variation across model settings less understood. To this end, we conduct an observational study of four ChatGPT Web settings from ChatGPT 5.2 to 5.5, covering Instant and Thinking modes at the URL, hostname, and entity levels. We examine how ChatGPT rewrites queries, retrieves candidate webpages, and selects references. The Instant settings remain mostly single-round, while GPT-5.4 Thinking frequently performs multiple retrieval rounds. Across repeated runs, candidates retrieved more often are also more likely to be cited. We then evaluate Google Search and the OpenAI Responses API as external proxies. Responses API candidates provide broader coverage, while its citations more closely match sources cited by ChatGPT Web. Both proxies match ChatGPT Web better at the hostname and entity levels than at the exact-URL level. Even when combined, they recover only about one quarter of cited URLs. We further test whether these external signals remain predictive when combined. Within the external candidate set, a supervised model combining them reaches AUCs above 0.74 for GPT-5.3 and GPT-5.5 at all three levels. These findings suggest that GEO should distinguish retrieval visibility from citation visibility, account for rewritten queries and run-to-run variation, and consider hostname- and entity-level presence in addition to exact URLs. Differences across settings also suggest that GEO strategies should be re-evaluated when the deployed setting or serving environment changes.