Chatbot performance scores vary significantly between api and interface use
API Benchmark Scores Do Not Reliably Transfer to Chatbot Interfaces
Artificial IntelligenceSoftware Engineering
Summary
Models like ChatGPT are often judged by scores from APIs, but these scores don’t always match how the models perform when used in real chatbot interfaces. The authors tested several major models and found that the API versions generally look better in accuracy and consistency than the same models accessed through chatbot interfaces. They also tried adjusting settings in the API to mimic the interface behavior, but this didn’t fully close the difference. This means benchmark results from APIs might not reliably show how well a chatbot will actually work for users.
What this means in practice
- •For machine learning engineers: Evaluate chatbot models directly in their user-facing interfaces instead of relying solely on API benchmark scores to better predict real-world performance.
- •For ai product managers: Make more informed decisions about deploying AI chatbots by recognizing that API performance metrics may overstate actual user experience quality.
Authors
Jennifer Wang, Joachim Baumann, Daniel E. Ho, Sanmi Koyejo
Abstract
Benchmark scores are a central currency in model releases: they inform purchasing decisions, shape public trust, and influence policy. Yet, a key assumption underlying benchmark scores is that the model performance measured through APIs faithfully reflects the behavior of deployed systems. We challenge this assumption by auditing ChatGPT, Claude, and Gemini across seven systems and nine benchmarks spanning general capability, social bias, and sycophancy. We find systematic API--interface differences in both accuracy and consistency. On average, API evaluations score 3.4 percentage points higher in accuracy and 2.1 percentage points higher in test--retest agreement than corresponding interface evaluations. For ChatGPT, the performance difference between API and interface access exceeds the API-only difference between GPT 5.3 and GPT 5.4. Put differently, switching access surfaces can degrade performance as much as downgrading a full model generation. We further test whether exposed API controls can reproduce interface behavior by varying system prompts, sampling parameters, and reasoning settings. These controls shift behavior in some cases but do not reliably eliminate the gap. Our findings document a context-validity gap: measurements obtained through APIs do not necessarily generalize to corresponding deployed interfaces, complicating the use of API evaluations as proxies for deployed systems.