Evidence-in-the-Loop: Trace-Driven Optimization for Customer-Service LLM Agents
2026-07-20 • Information Retrieval
Information Retrieval
AI summaryⓘ
The authors describe a customer-service chatbot system designed to improve answer quality without breaking rules or ignoring human help needs. They combine several methods to find relevant FAQ answers from different sources and carefully mix this with policy rules and conversation context using a fixed workflow. Their approach includes ways to track errors and improve how the system retrieves and ranks answers over time. These methods aim to keep the chatbot reliable and easy to audit in real-world use.
customer-service botslarge language modelsBM25retrieval-augmented generation (RAG)rerankingLangGraph DAGFAQ retrievalpolicy rulesfine-tuningerror diagnosis
Authors
Chunming Wu, Dafei Qiu, Congde Yuan, Charles Quan, Jun Wu, Suipeng Li, Mo Wu, Gavin Xie, Hope Chen, Max Yao
Abstract
Production customer-service bots must improve answer quality across iterative releases, yet large language models must not bypass evidence boundaries, policy rules, or human-handoff safeguards. We present an \textbf{Evidence-Grounded Customer-Service Agent Workflow} deployed in a real-world customer-service setting. BM25 recall, issue-title-vector recall, issue-description-vector recall, weighted RRF fusion, and cross-encoder reranking construct grounded FAQ evidence for controlled LLM decisions. Policy-guided orchestration then combines this RAG evidence with scenario-specific rule evidence, conversation memory, and clarification state inside a fixed LangGraph DAG~\cite{langgraph2024}. The paper contributes three reusable deployment patterns: \textbf{hybrid RAG evidence construction}, where multi-channel retrieval and reranking produce auditable FAQ candidates; \textbf{evidence-grounded issue/action decision}, where an Evidence-Grounded Decision Module selects an issue/action from typed FAQ evidence and scenario-specific rule evidence; and \textbf{trace-driven RAG and reranker improvement}, where traces diagnose whether failures come from recall, ranking, final candidate selection, clarification, rule-derived evidence, or action policy, and where reranker fine-tuning is evaluated not only for in-domain gain but also for forgetting risk.