Contract memory compiler improves multi-step question answering accuracy
Contract Memory Compiler: Resolve, Then Traverse
Artificial Intelligence
Summary
Answering questions that need many steps and rely on long histories is hard. The authors show that when the information changes, just grabbing recent updates is not enough — you need to find the right original details based on the updated facts. They built a system called Contract Memory Compiler (CMC) that looks at all changes first, figures out which pieces of the entire history really matter, and then gives only those details to the answer engine. This approach works better than older methods and leads to more accurate multi-hop question answering.
What this means in practice
- •For chatbot developers: Enable chatbots to answer complex questions using long and updated conversation histories by selecting precise evidence after resolving all updates.
- •For knowledge base engineers: Improve retrieval accuracy in systems tracking evolving entities by referencing original records identified through updated relations before question answering.
Authors
Zhi Song, XiMing Xing, Chunhan Li, Weian Mao, Zhenchao Tang, Hanbo Huang, Fan Xu, Jiale Zhou, Jiahui Guan, Zejian Ding, Chen Ma, Lusheng Wang
Abstract
External memory lets language-model agents answer questions about histories too long for the answer model's context window. Updates create a harder problem than retrieving a recent fact: changing one relation can redirect a multi-hop question to records about an entity absent from the question. We study this update-dependent evidence selection problem and introduce the Contract Memory Compiler (CMC). Before seeing a question, CMC uses a language model to identify relations in the history and record where each one was stated. It applies later updates to determine the current relations, follows them from entities named in the question, and passes the corresponding original records to the answer model in one call. Thus the current state determines which evidence is read, rather than merely refreshing values in a previously selected context. To the best of our knowledge, CMC achieves state-of-the-art multi-hop accuracy on FactConsolidation, reaching 78.25% overall and 61.0% at 262K. With the extracted relations and answer model held fixed, selecting evidence before resolving updates reduces multi-hop accuracy to 21.50%. We also introduce MQuAKE-MemStream, a derived dataset of ordered memory streams built from MQuAKE-Remastered counterfactual cases.