Financial agents struggle with multi-step search queries using exact intermediate checks
FinHopper: A Progressive Multi-Hop Benchmark for Complex Financial Agents with Step-wise Evaluation
Machine Learning
Summary
Financial agents need to answer complex questions by looking up multiple facts step-by-step, where each step depends exactly on the previous answer. The authors created FinHopper, a test set that measures how well these agents handle multi-step financial questions up to six steps long, checking progress after each step. They found that even advanced language models drop sharply in accuracy as the steps increase and often make errors that look plausible but are wrong. Their work helps understand the challenges in building financial question-answering systems that reason carefully over multiple steps.
What this means in practice
- •For financial software developers: Develop and test financial question-answering systems that must follow strict multi-step logic using exact intermediate answers up to six steps.
- •For business intelligence teams: Evaluate and improve tools that provide answers from financial data where each step depends precisely on previous findings to reduce cascading errors.
Authors
Yuxuan Wu, Miaopei Lin, Qiufang Ying, Lin Dan, Hao Zhao, Qirui Liu, Yuteng Shen, Shaohui Wu, Weihong Luo, Xiku Du, Xing Tang, Hao Chen
Abstract
Financial agents are expected to answer factual investment queries through open search, but financial multi-hop retrieval differs from general multi-hop QA in a crucial way: each intermediate answer is an exact typed constraint, such as an entity, date, event, or numerical indicator, that determines the next query. This strict answer-conditioned dependency makes small intermediate errors cascade into plausible but wrong search trajectories. We introduce FinHopper, a benchmark for complex financial agents with controlled reasoning depth from 1 to 6 hops, covering stocks, funds, and macroeconomic facts, with gold intermediate answers for process-level evaluation. To construct FinHopper at scale, we design a state-transition-based synthesis framework that composes verified atomic financial QA patterns into dependency-preserving chains, followed by targeted in-synthesis verification, model-ensemble checking, and public-search validation by financial experts. Experiments with 20 LLM-agent configurations show that public search tools are necessary but insufficient: performance drops sharply as hop depth increases, the best model remains far below human experts, and process-aware scoring reveals partial reasoning progress hidden by final-answer-only evaluation. The benchmark is available at https://github.com/justajustin/FinHopper/.