CTRAG: An In-Context Retrieval-based Framework for Automated Compliance Checking using LLMs
2026-08-03 • Computation and Language
Computation and Language
AI summaryⓘ
The authors created CTRAG, an automated system that helps check if companies are following important rules, especially in tricky cases involving third-party services like cloud providers. CTRAG uses smart techniques to find relevant questions from regulations and compares them with company documents to see if the rules are met. Their tests show that CTRAG is quite accurate and catches most problems, making compliance checking faster and less error-prone. They also tested it for real in a major consulting firm, showing it helps reduce manual work and improves trust in regulatory processes.
Regulatory complianceRetrieval-Augmented Generationadaptive chunkingin-context learningthird-party servicescompliance verificationF1-scorerecallBig Four firmsautomated compliance checking
Authors
Muhammad Roman, Karen Rafferty, Barry Devereux
Abstract
Trust is fundamental in modern regulatory ecosystems, and compliance checking plays a critical role in fostering that trust. Regulatory compliance verification is essential for businesses operating in highly controlled environments, as it ensures alignment with sector-specific guidelines across domains such as financial reporting, data privacy, and cybersecurity. Manual compliance testing, however, is often time-intensive and prone to inconsistencies, particularly when compliance depends indirectly on third-party services such as cloud providers, where vendors rely on external providers to meet regulatory standards. In this paper, we present CTRAG, a novel Retrieval-Augmented Generation (RAG) pipeline designed for automated compliance checking. CTRAG employs advanced strategies, including adaptive chunking, dynamic retrieval configurations, and in-context learning, to improve the precision and relevance of compliance assessments. By extracting control questions from regulatory texts and cross-referencing them with unstructured company documentation, CTRAG achieves highly accurate, document-informed compliance verification, even in cases of indirect compliance through third-party services. Empirical evaluations demonstrate significant improvements, with CTRAG achieving an F1-score of 78% and a recall of 85% in the final deployed configuration, ensuring minimal missed non-compliance cases while reducing manual reviewer effort in a real-world deployment. To validate CTRAG value, we developed and deployed a POC within a Big Four professional services firm, applying it to real-world cases and cross-checking results against manual compliance reports. These findings highlight CTRAG potential to streamline compliance workflows, mitigate risks, and enhance regulatory trust in complex, high-stakes environments.