Financial audit tasks benchmark created using private real data insights
FinancialAuditBench: Benchmark Construction under Differential Privacy Using Real-World Priors
Artificial Intelligence
Summary
Measuring how well AI can perform financial audit tasks is hard because real audit data is sensitive and private. The authors created FinancialAuditBench, a set of realistic audit tasks generated using privacy-protected statistics from actual audits combined with expert knowledge. They tested eleven advanced AI models on these tasks and found that while AIs can handle many routine audit steps, they sometimes make mistakes or follow wrong procedures. This approach can help better assess AI in finance and other sensitive areas without exposing confidential data.
What this means in practice
- •For financial technology teams: Evaluate AI tools on realistic audit tasks generated without exposing real client data to improve tool reliability and compliance.
- •For privacy teams in enterprises: Use the framework to create synthetic data-driven tasks for validating AI models while preserving sensitive information across various domains.
Authors
Jerry Huang, Sarvesh Babu, Matt Van Buren, Alexander Wang, Pranav Pillai, Arush Jain, James P. Burton, Julia Hockenmaier
Abstract
As AI agents are becoming widely adopted in the financial services industry, careful measurement is essential to understand where they can be reliably deployed and where oversight and professional review remain necessary. Such measurement, however, is constrained by limited access to proprietary or privacy-sensitive data. Existing benchmarks therefore often rely on publicly available data, human- and/or LLM-authored tasks, or simplified settings. We introduce FinancialAuditBench, a benchmark for evaluating agents on financial statement audit tasks, along with a framework for systematically generating synthetic engagements. Our task generation framework leverages differentially private aggregate statistics from historical audits along with audit expertise contributed through over 1,100 hours of benchmark development and review. FinancialAuditBench consists of 90 tasks spanning workpaper completion and review across six synthetic audit engagements, each containing an average of 179 files. Evaluation on eleven frontier models shows that while agents complete substantial portions of staff-level audit tasks well, they sometimes perform inappropriate procedures or produce incorrect documentation. Beyond financial auditing, our framework offers an approach for systematically generating synthetic tasks for model evaluation and training in privacy-sensitive domains.