Context infrastructure reduces token use and latency in information retrieval loops
Context as Infrastructure: Knowledge Compilation for Retrieval in Agentic Systems with Harnesses
Information RetrievalArtificial IntelligenceComputation and Language
Summary
The paper studies how computers that gather and use information through repeated web searches can become faster and more efficient. The authors find that organizing and enriching information before searching—called knowledge compilation—does not directly improve accuracy but greatly reduces the amount of information the system must process and speeds up response time. They built a system called RAGFlow that breaks down the search and decision process into clear steps, giving the computer just the necessary information instead of many documents. This approach saves time and helps the system work more smoothly even when handling complex searches.
What this means in practice
- •For software engineers building ai assistants: Build AI assistants that efficiently handle repeated web searches by organizing retrieved information into minimal, enriched context sets, reducing token use and response times.
- •For enterprise knowledge management teams: Integrate structured contextual data sources like claim-evidence trees and timelines into enterprise search to improve information governance and onboarding experiences.
- •For customer support platform developers: Develop customer support tools that use compiled contextual knowledge to speed up retrieval of relevant information during multi-turn conversations.$Commercial implications: Enables faster, context-aware support systems that reduce wait times and improve answer relevance, making commercial support products more competitive.
Authors
Zhichang Yu, Baining Cao, Sbaaoui Idriss, Qi Wang, Gangqiang Xu, Ling Qin, Xiaoling Jin, An Liu, Shaohao Cai, Xiaoyang Bao, Shengle Chen, Changchang Ding, Pengfei Zhu, Jingxiang Xu, Yifeng Mao, Lu Liu, Hai Jin, Yingfeng Zhang
Abstract
Retrieval-augmented generation is becoming agentic. A search harness writes queries, reads results, reformulates, and decides when to stop. This largely solves accuracy: a harness with only BM25/grep and MiniMax M3 answers 85% of BrowseComp-Plus and 91% on multi-hop FRAMES in our runs, without compiled or vector indexes. It exposes a context problem: an undisciplined loop repeats queries, rereads documents, and fills the window with unused passages. The missing layer is knowledge compilation: a configurable ingestion pipeline where LLMs enrich documents with context no chunk contains alone--navigation hierarchies with claim-evidence leaves, entity graphs, timelines, mind maps, ontologies, wikis, and memory. Compilation is not a better retriever: under a competent harness, structural artifacts gave no detectable accuracy gain on FRAMES, and a PageIndex-style outline degraded at scale. It is a supply layer. A navigation tree whose leaves are claim-evidence pairs--the mechanism we measure--cuts query tokens by 91% when evidence suffices and adds 1.9% when it does not, a 38% average reduction; this implies a 2% break-even sufficiency rate and a measured rate near 43%. A decision gate cuts latency from 140 s to 30 s, though substitution's token saving is bought with retrieval rounds and carries its own latency cost. Other artifacts--ontology, graph, wiki, timeline, mind map, memory--pay off in deployment (governance, integration, onboarding, temporal orientation, continuity) where accuracy benchmarks cannot see them. We describe RAGFlow's pipeline and harness (open source at https://github.com/infiniflow/ragflow), which turns the loop into five disciplined stations and keeps the context entering the model the smallest sufficient set; together they form an agent-native database: a store that hands the agent compiled, governed context instead of a ranked list of passages.