Multi-hop reasoning needs checks beyond correct answer accuracy
REALHOP: Rethinking Multi-Hop Reasoning Evaluation via Behavioral Auditing
Artificial Intelligence
Summary
Answering complex questions often takes several steps, connecting different pieces of information from various places. The paper shows that just getting the right answer doesn't mean a model truly followed all the intended reasoning steps; sometimes models rely on shortcuts or memorized facts instead. The authors propose a new method called REALHOP, which tests whether every reasoning step really matters by removing key evidence and seeing if the model's answer changes. Their approach reveals that models often don't depend on all steps but can be guided to do so, helping make evaluations of multi-step reasoning more reliable.
What this means in practice
- •For natural language processing engineers: Improve multi-hop question answering systems by verifying each reasoning step’s importance, leading to more trustworthy model responses.
- •For ai system testers: Develop evaluation tools that detect whether AI models truly use intended multi-step reasoning, improving testing accuracy in complex question answering.
Authors
Jiawen Tao, Xiaokun Yuan, Yaoming Li, Chenxu Liu, Mengzhou Wu, Tong Yang, Maxm Pan
Abstract
Complex questions often require multi-hop reasoning that connects facts distributed across sources or distant regions of a long context through intermediate steps. Benchmarks commonly evaluate this ability with questions built around predefined reasoning chains, treating a correct answer as evidence that the intended composition was used. Yet answer correctness alone leaves open whether success depends on the evidence associated with each intended step: models may instead rely on memorized associations, shorter paths, or partial evidence. We examine this dependence using the Behavioral Necessity Rate (BNR), which measures how often targeted evidence removal prevents answer recovery on initially correct instances. Across five existing benchmarks, panel-mean BNR ranges from 16.6% to 48.9%, exposing a substantial gap between annotated structure and observed dependence. Guided by this diagnosis, we introduce REALHOP, a diagnose-construct-verify framework that rebinds entities, factorizes selected relations, adds complete competing paths, and places evidence at traceable locations. Structural and semantic checks precede freezing; behavioral interventions follow. On 790 paired MuSiQue questions, REALHOP raises panel-mean BNR from 27.4% to 94.4% while retaining high Full accuracy. It also yields high BNR on REALHOP-FRAMES and REALHOP-LONGBENCH. On 216 long-context questions, the matched multiple-choice spread across 16 models grows from 13.9 to 59.2 points and persists under repeated open-ended evaluation. Together, these results show that a conceptually coherent chain and a correct final answer do not by themselves establish multi-hop reasoning. Verifying that success depends on every intended hop is therefore as fundamental to multi-hop evaluation as measuring answer accuracy itself.