Multi-Hop Knowledge Composition is Bound by Pretraining Exposure
2026-06-08 • Computation and Language
Computation and Language
AI summaryⓘ
The authors show that large language models can remember simple facts well but struggle to combine them to answer questions that need multiple steps, like figuring out when a friend's birthday is. They tested this by making sure some people were only seen alone during training, while others appeared in more complex contexts. Even when models knew individual facts perfectly, they failed to put them together for new people they hadn’t seen in combined situations. The authors found that seeing examples with combined facts during training is necessary for the models to succeed at reasoning across multiple steps.
Large Language ModelsImplicit multi-hop reasoningPretrainingCompositionalityData augmentationForward pass1-hop accuracyMemorizationContext exposureMulti-step reasoning
Authors
Yannis Karmim, Luis Marti, Djamé Seddah, Valentin Barrière
Abstract
Large Language Models fail at implicit multi-hop reasoning: a model answers "When was $X$ born?" and "Who is $Y$'s closest friend?" correctly but fails on "When was $Y$'s closest friend born?" in a single forward pass, even when both facts are perfectly memorized and individually retrievable. We study this failure in a controlled natural language setting with a strict separation between individuals exposed to compositional contexts during pretraining and those that never appear in any such context. We confirm that compositional failure persists even at 97% 1-hop accuracy, establishing the gap as a pretraining failure rather than a knowledge absence. We propose and test nine data-centric augmentation formats and find that compositional pretraining transfers to unseen questions for exposed individuals, but never to individuals absent from compositional pretraining, suggesting that exposure to compositional contexts during pretraining is a necessary condition for implicit multi-hop reasoning.