Multi-hop reasoning often sounds right locally but fails globally
Locally Sound, Globally Insufficient: The Local-Global Gap in Multi-Hop Reasoning
Computation and Language
Summary
Sometimes, a computer can follow reasoning steps that each seem correct on their own but still fail to answer a question properly. This problem, called the local-global gap, means the reasoning looks good step-by-step but doesn’t lead to the right overall answer. The authors studied many examples and found this happens frequently and is hard for current checks to detect. They proposed a new training method called E-Closure that helps the computer reason both correctly at each step and in the big picture, improving accuracy and reliability.
What this means in practice
- •For natural language processing teams: Improve training of AI models to generate reliable multi-step reasoning that truly answers complex questions.
- •For automated customer support teams: Enhance AI chatbots’ ability to provide accurate answers through consistent reasoning over multiple information sources.
Authors
Bohao Chu, Hendrik Damm, Qianli Wang, Hui Wang, Shuning Zhang, Christoph M. Friedrich, Norbert Fuhr
Abstract
Reliable multi-hop reasoning requires more than locally supported steps: a trace can be sound at every reasoning step yet still fail to answer the question as a whole. We call this failure regime the local-global gap (LGG), in which the trace is locally sound yet globally insufficient. Local soundness requires each step to be supported by the available evidence and preceding steps, whereas global sufficiency requires the reasoning trace to align with the question and establish the submitted answer. In a human-adjudicated diagnostic of 2,598 responses across three multi-hop QA benchmarks and three models, we find that the LGG occurs in every benchmark-model combination and accounts for nearly half of globally insufficient responses overall. However, conventional faithfulness verifiers that check claims against the evidence largely miss these failures: at thresholds retaining at least 95% of reliable traces, recall for LGG cases is substantially lower than that for locally unsound traces. To address these failures, we formalize three dependencies for reliable reasoning: evidence-to-step support, question-to-trace alignment, and trace-to-answer closure. Instead of post-hoc diagnosis, we introduce E-Closure to supervise these dependencies during training, combining generation supervision on supported original and counterfactual responses with bidirectional switching constraints. Averaged over three benchmarks and three backbones, existing fine-tuning baselines improve accuracy and local soundness over the base models, but at the cost of global sufficiency. E-Closure improves both: among all fine-tuned methods, it achieves the highest average accuracy (92.8%) and trace reliability (89.0%) while yielding the lowest LGG rate (6.2%).