Large language models struggle to reason with changing conflicting information
Where and Why Search-Augmented LLMs Fail When Reasoning over Rapidly Evolving Knowledge
Computation and LanguageArtificial IntelligenceInformation RetrievalMachine Learning
Summary
As new information is published every day, it can be hard for AI models to figure out what is still true, especially when old and new facts conflict. The authors created a new set of tests and tools called TREK to understand where these AI models go wrong. They found that the biggest problem isn’t finding the right information, but that the AI often gets confused when reasoning about conflicting or outdated details after it gathers that information. Their work helps developers better diagnose and improve AI reasoning over fast-changing knowledge.
What this means in practice
- •For ai developers: Diagnose and improve reasoning abilities of language models on questions involving changing or conflicting information using the TREK framework and benchmarks.
- •For digital archivists: Evaluate the reliability of AI tools for interpreting and updating evolving digital records affected by conflicting or outdated information.
Authors
Tianyi Hu, Bhavyajeet Singh, Niket Tandon, Akhil Arora
Abstract
As the world changes, new information is published while older records remain online. This complicates reasoning for large language models (LLMs), which must retrieve evidence and resolve conflicts to determine which information remains valid. Existing evaluations treat information seeking as a black box, revealing little about where and why LLMs fail. To address this gap, we introduce TREK, a suite of three components: TREKTRACE is a plug-and-play framework that attributes failures to specific stages of retrieval and reasoning. We apply it to TREKBENCH-WILD, a meta-benchmark of real-world questions that require complex reasoning over rapidly evolving knowledge. Surprisingly, the main bottleneck is post-retrieval reasoning rather than retrieval: frontier models are often misled by conflicting evidence even after retrieving all the information needed to answer. To study this bottleneck, we build TREKBENCH-CTRL, a controlled benchmark with distractors spanning temporality, relevance, and trustworthiness, and show that benchmarks dominated by outdated distractors substantially underestimate task difficulty. Together, TREK provides a testbed for diagnosing and improving reasoning over rapidly evolving knowledge.