Benchmark reveals compliance gaps in AI customer service workflows
CXBench: A Benchmark for Process-Compliant LLM Agents in Regulated Customer-Experience Workflows
Artificial Intelligence
Summary
Many AI agents can successfully complete customer service tasks, but they often skip important steps like asking for permission or sharing legal information, which can cause problems in regulated industries. The authors created cxbench, a benchmark that separately checks if AI agents follow proper procedures as well as if they get the right results. Testing several AI models, they found most passed when only final results were checked but failed when process rules mattered. This shows it is important to evaluate AI on how they do their work, not just the end outcome.
What this means in practice
- •For contact center developers: Assess and improve AI agent adherence to legal and procedural rules in customer support workflows using cxbench's detailed compliance criteria.
- •For compliance teams: Use cxbench to audit AI-driven customer interactions for required disclosures and consent steps to ensure regulatory requirements are met.
Authors
Boxi Yu, Yuzhong Zhang, Zhenfeng Su, Linfu Xi, Yinghan Yao, Shitao Chen, Yang Cao, Liting Lin, Lionel Briand, Emir Muñoz
Abstract
Tool-using language agents increasingly perform regulated customer-service workflows, where a correct final state is insufficient: agents must also provide disclosures, obtain consent, and follow required procedures. Existing tool-agent benchmarks largely score final business states, accepting trajectories that reach correct outcomes through non-compliant processes. We introduce cxbench, a benchmark that evaluates outcome and process compliance separately across six regulated domains. Each task combines a customer request, a tool-based environment, and explicit process requirements. Rule cards link cited regulatory obligations to observable trace evidence. A hybrid grader combines deterministic checks for ordering and authority with an LLM judge for semantic requirements. We define a false-pass as an outcome-pass run that fails process checks. Across seven models and 120 tasks, 55.5% of the 840 retained executions are false-passes, representing 80.1% of outcome-passing runs; only 13.8% pass both checks. Process-inclusive evaluation changes the observed model ordering and exposes differences hidden by outcome-only scoring. cxbench makes these failures inspectable through trace-linked criteria.