Financial question answering needs better ambiguity detection and clarification
FinInteract: Benchmarking Clarification and Intent Integration in Ambiguous Financial Question Answering
Artificial Intelligence
Summary
Financial questions often have multiple valid answers depending on how you interpret them, but current AI models usually guess the most common meaning instead of asking for clarification. The authors created FinInteract, a benchmark to test whether AI can recognize ambiguous financial questions, ask the right clarifying questions, and then give correct answers. They found that AI models like GPT-4o answer well when given clear context but struggle when they must figure out the right meaning on their own. This work shows the importance of teaching AI to clarify ambiguous financial queries before answering.
What this means in practice
- •For financial software developers: Build AI assistants that clarify ambiguous financial queries to improve answer accuracy from regulatory filing data.
- •For customer support teams in finance: Enhance chatbots to ask clarifying questions before answering financial questions, reducing errors and confusion.
Authors
Xinyu Wang, Tung Sum Thomas Kwok, Zhenghan Tai, Guang Cheng
Abstract
Large language model agents increasingly answer financial questions by searching regulatory filings. Such questions are often deceptively under-specified: Meta Platforms' "operating income" is $46.75B consolidated but $62.87B for the Family of Apps segment, and each reading is exactly verifiable against the filing. A capable agent should recognize the ambiguity and ask, rather than commit to a plausible but unintended reading. Existing financial benchmarks cannot measure this, because one gold answer per question cannot separate agents that resolve the ambiguity from those that guess the common reading, a blind spot we call the single-gold illusion. We release FinInteract, a bilingual (English/Chinese) benchmark of 173 instances that pairs each question with a default and an intended interpretation across a five-category ambiguity taxonomy, and grades whether an agent elicits the right clarification and then integrates it. Re-grading identical outputs against the default rather than the intended reading inflates GPT-4o's accuracy by 3.1 times, confirming the illusion. Beyond it, we find that models answer above 90% once the interpretation is supplied but at most 28.9% when they must elicit it themselves, that targeting is uneven across a taxonomy well powered for entity scope and metric definition and exploratory elsewhere, and that conditioning on the ambiguity category improves resolution at both inference and training time.