Agentic rule discovery improves auditing of data quality rules
AuDQ: Auditable Data Quality through Agentic Rule Discovery
Artificial Intelligence
Summary
Keeping track of data quality rules in important systems is tricky because the checks are scattered and hard to update or review. The authors suggest making these rules more organized, linked to the policies they follow, and easy to replay to see how they were applied. They built a tool that automatically finds these rules and allows humans to review them. Their initial tests show this automated approach can produce rules as good and varied as those made by experts in finance and retail data.
What this means in practice
- •For data governance teams: Automate discovery and review of data quality rules linked to governance policies for better auditing and compliance tracking.
- •For financial data analysts: Use autonomous rule discovery to maintain and update quality checks on financial datasets without relying solely on manual expert input.
Authors
Jacopo Biggiogera, Nicolas Colombo, Krati Saxena, Bhaskarjit Sarmah, Stefano Pasquali
Abstract
High-stakes tabular pipelines rely on scattered, static quality checks that are hard to evolve and audit against policy intent. Regulated domains particularly suffer as the LLM-era automation raises expectations for traceability and accountability. We propose a vision for an ontology-grounded, repository-native data quality layer in which rules are first-class artifacts: versioned, provenance-linked to governance texts, and re-playable through immutable execution traces. As a first step toward validating this vision, we present a concrete implementation of its agentic rule-discovery core with two rule acquisition modes (Full-Agentic and Guided) and a human-in-the-loop (HITL) review interface. We define an evaluation framework and run experiments addressing whether our fully autonomous, divide-and-conquer agentic rule discovery produces rules of the same quality and diversity as those specified by a human expert, reporting preliminary results on two real world industry datasets in the finance and retail domains.