Probabilistic checks improve financial citation role accuracy beyond just number matching
Same-Number Citation Swaps: Stress-Testing Jev as a Financial Evidence Judge
Computation and Language
Summary
Financial reports often reuse the same numbers in different places, which can confuse systems that check if calculations are correct. The authors study how well a system called Jev can verify if a number cited in a financial calculation supports the correct financial role, not just if the number matches. They find that while it’s easy to check if numbers match across cites, distinguishing the exact financial meaning or role behind the number is harder. Their controlled tests help measure how probabilistic verification can detect wrong citations even when numbers match exactly.
What this means in practice
- •For financial software developers: Improve automated tools to check if financial calculations cite the correct financial data roles, not just matching the numbers.
- •For audit teams: Help audits by flagging citations in financial documents that may reference numbers with incorrect financial meanings, enhancing report reliability checks.
Authors
Chuhong Xu, Bo Su, Ziyao Chen, Ruiyang Xu, Shimeng Dai, Xinyu Qiu
Abstract
Financial reports repeat values across periods, metrics and accounting lines, allowing an LLM-generated calculation to be numerically correct while citing the wrong financial role. We evaluate what probabilistic evidence verification adds beyond number matching using Jev as a source-support verifier for GPT-4.1-mini calculation traces. A signed-number-at-pointer baseline explains most recovery over exact quotation checks. To isolate the remaining role-recognition problem, we hold operands and arithmetic fixed, move citations between same-number cells, and retain controls that express equivalent facts. These contrasts reveal both wrong-role citations that pass and valid alternative citations that are withheld. Explicit column labels improve selected wrong-role decisions while also lowering support for some equivalent evidence. A constructed follow-up on 36 new source pages, labeled by a non-author reviewer, extends this evaluation and exposes the same tradeoff between detecting role errors and retaining valid citations. The contribution is a controlled evaluation that identifies what a probabilistic financial verifier distinguishes when numerical matching is held fixed. For LLM-based financial assistants, it makes numerical correctness, cited-role support and acceptance outcomes separately assessable.