Sales agent trained on real chats competes well with humans and AI
Selling in the Wild: Benchmark and Randomized Field Evidence for a Sales Agent Trained on Commercial Conversations
Information RetrievalComputers and Society
Summary
Selling products online often involves chatting with customers, either by humans or automated agents. The authors tested Javo, an AI sales agent trained on millions of real sales conversations, across public benchmarks and live websites. Javo increased sales revenue by 10% compared to having no agent and outperformed a general AI model by over 20% in converting visitors to buyers. When compared to human sales associates, Javo performed similarly while costing far less per conversation. It also maintained safe customer service standards.
What this means in practice
- •For e-commerce businesses: Deploy AI sales agents trained on real customer chat data to increase sales revenue and reduce sales costs compared to human associates.$Commercial implications: Provides a cost-effective AI sales agent capable of integrating into live sales chats and boosting conversion rates across multiple industries.
- •For customer service platform developers: Integrate specialized AI agents fine-tuned on authentic sales dialogues to improve buyer engagement and conversion metrics.
Authors
Benjamin Tannenbaum
Abstract
We test whether Javo, a sales agent fine-tuned on 4.6 million consented, de-identified commercial conversations, sells better than a frontier general-purpose model and better than human sales associates. The evaluation has four parts. Study 1 scores the agent on two public sales benchmarks, SalesLLM and SmartSalesBench, against 10 frontier and open-weight models. Study 2 is a randomized controlled trial on 100 live websites in 14 industries (e-commerce, travel and hospitality, and high-consideration services), run from 1 May to 30 September 2026, in which 14.8 million visitors were randomized to no agent, Javo, or a frontier-model agent with an identical interface, catalog tools and guardrails. Study 3 isolates the effect of the model by comparing those two agent arms. Study 4 compares Javo with each site's own human associates on 20 sites with staffed live chat. Relative to no agent, Javo changed revenue per visitor by +10.0% (95% CI +5.1 to +14.9). Among engaged visitors, it changed conversion by +21.6% relative to the frontier-model agent (95% CI +16.9 to +26.3). Against human associates, its engaged conversion was 11.26% versus 11.20% (non-inferior, margin -7%, one-sided p = 0.004; no significant superiority), at 1.8% of the cost per conversation. Return rates, unsupported product claims and complaint rates met the prespecified safety guardrails.