NumTrace audits financial AI agents for correct number sources and meaning
NumTrace: Auditing the Number Provenance of Financial Analyst Agents
Computation and LanguageArtificial Intelligence
Summary
Sometimes financial AI tools give numbers that look right but are actually taken from the wrong place or mean something different. The authors created NumTrace, a test with many questions that checks not just if a number is correct but also if it comes from the right financial document section and means the right thing. This helps find mistakes where the number matches but refers to the wrong account or period. They tested some popular AI agents and found most mistakes were these 'right number, wrong account' errors.
What this means in practice
- •For financial ai developers: Improve AI tools by auditing whether generated numbers are from the correct financial sources and contexts using NumTrace's multi-verdict approach.
- •For financial compliance teams: Verify AI-generated financial claims with detailed checks on number origin and meaning to reduce errors in reporting and auditing.
Authors
Tianlei Zhu, Matt White, Xue Liu, Xiao-Yang Liu
Abstract
A numerical claim generated by a financial analyst agent can match a number in a company filing but refer to the wrong entity, concept, period, unit or scope. A check based on reference answers can mark such an answer as wrong but does not explain why it fails. In this paper, we present NumTrace, a new benchmark that audits the provenance of each numerical claim made by financial analyst agents: its source and financial semantics. NumTrace currently has 540 questions over 2,163 SEC EDGAR documents. To the best of our knowledge, it is the first numerical benchmark about who holds a company's shares (Form 13F) and what its executives are owed (Item 402(i)). On a frozen snapshot of these documents, we build the reference answers according to accounting rules. NumTrace then gives each claim three separate verdicts: value, binding and support. By retaining separate value and binding verdicts, NumTrace detects right number, wrong account (RNWA) errors: a claim has an RNWA error when it passes the value check but fails the binding check. The RNWA rate quantifies the gap between value-only and joint value-and-binding pass rates. Finally, we audit three agents, Claude Opus 5.5, GPT-6 Astra and DeepSeek-V4.1-Flash. RNWA errors make up 67.5-81.8% of each agent's failing claims. In particular, on 13F co-filing group questions, about 91% of claims pass the value check, but fewer than half pass the binding check.