Simple agentic retrieval aids customer support workflow automation
Frustratingly Easy Agentic RAG via Portable Skills: A Case Study in Customer Support
Artificial Intelligence
Summary
Customer support specialists need detailed instructions from lots of policy documents to solve cases properly. The authors show that instead of using complicated systems to find the right policies, a simpler method works well. They built a tool that uses clear instructions, a way to look up information, and a well-organized summary of documents, which helps create good step-by-step solutions. This easy system reached over 99% match with expert solutions and uses less engineering effort. Although slower than some production tools, it is good enough when speed is less important and people double-check the answers.
What this means in practice
- •For customer support teams: Generate accurate case resolution steps from rich policy documents with minimal custom engineering overhead.
- •For enterprise automation engineers: Deploy portable, easy-to-maintain retrieval-based skills for complex document-driven workflows without retraining models.
Authors
Istok Bako, Bonan Min, Tiffany Shao, Claire Cheng, Yashar Mehdad
Abstract
Enterprise workflow automation typically requires retrieving a complete and accurate set of relevant information from large operational document corpora (SOPs, policy documents) given a specific scenario or request. In customer support (CS), for example, human specialists must determine the correct sequence of resolution steps for each case, a task that demands precise grounding in policy documentation. Meeting this bar in production typically requires complex document chunking, indexing, and retrieval pipelines, often layered with learned re-rankers, query rewriters, and bespoke agentic RAG infrastructure. We report a deployment case study of a simpler alternative: a portable skill consisting of a natural-language instruction, lightweight tools (a retrieval API, format-checking scripts), and a corpus enriched with an LLM-generated navigation layer, which is a structured "Wikipedia-style" index over the underlying documents. This agentic RAG approach is straightforward to implement (1-2 engineer days), portable across capable agentic harnesses (e.g., Claude Code), and performs strongly on the challenging task of CS resolution workflow generation, achieving 99.2% procedural equivalence to human-approved ground truth and 83.3% evidence sufficiency. It is also roughly an order of magnitude slower per case than our deployed production assistant (~90 s vs. ~8-9 s), so we claim no latency advantage. Our claim is scoped to that setting: a mature, well-curated corpus, a latency-tolerant workflow, and human review of every case. There, the skill did the work of a complex RAG stack, and maintenance became a matter of editing text and data rather than retraining models.