Transaction question answering reveals key challenges in fraud detection
TxnQA: Benchmarking Analytical Reasoning and Evidence Grounding over Transaction Event Sequences
Artificial IntelligenceComputation and LanguageMachine Learning
Summary
Understanding patterns in transaction records is hard because they happen irregularly and need complicated calculations. The authors created a test called TxnQA that checks how well computer models can answer questions and explain their reasoning about transaction data, including spotting fraud risk. They found that while models can often give the correct final answer, they struggle to follow output rules precisely, do detailed calculations, and correctly show the evidence behind their decisions. These problems highlight important hurdles in making reliable transaction analysis tools.
What this means in practice
- •For financial fraud detection teams: Improve evaluation methods for AI tools that analyze transaction sequences and provide detailed fraud risk explanations.
- •For bank data analysts: Benchmark and enhance AI systems to ensure transaction event analysis includes accurate calculations and clear evidence attribution.
Authors
Rabimba Karanjai, Yiming Bian, Yun-Shiuan Chuang, Jenna L. Ballard, Zijie Pan, Shivani Shekhar, Yixi Li, Yirou Ge, Avinash Thangali, Richard Williamson, Linsey Pang, Prakhar Mehrotra
Abstract
Analyzing merchant transaction histories requires dynamic temporal filtering, entity resolution, and aggregation over irregular event streams--operations poorly captured by static tabular or time-series forecasting benchmarks. We introduce TxnQA, an analytical QA benchmark comprising 27 task families (81 templates) alongside a 5-scenario fraud-risk reasoning suite that evaluates both verdicts and supporting evidence over synthetic, privacy-mitigated transaction data. Ground-truth answers are computed programmatically, and model performance is decomposed into format-following rate (FFR), conditional content accuracy (CA), and end-to-end correctness (E2E). Evaluating prompted proprietary baselines and six LoRA-adapted open-weight models shows that adaptation reliably enforces output schemas but leaves arithmetic bottlenecks: Gemma-4-31B-IT and Qwen3-8B attain E2E scores of 0.867 and 0.816, whereas evaluated MoE checkpoints reach only 0.632-0.713. Furthermore, on the 500-instance fraud suite, verdict accuracy frequently exceeds 80%, yet joint accuracy (requiring exact evidence attribution) collapses to at most ~23%. These findings identify schema compliance, semantic calculation, and evidence attribution as three distinct bottlenecks in transaction-sequence QA.