Index improves itself to boost search across many documents
Self-Evolving Search Index
Information RetrievalArtificial Intelligence
Summary
Finding relevant information in lots of documents depends on how well the search index represents each document. The authors noticed that current indexes do not adapt well to changing search needs without human help. They created SELF-INDEX, a system that can automatically detect where the search is failing, fix the parts of the index that cause problems, and test these fixes. It also tries out new search queries to keep improving. Their tests show that SELF-INDEX works better than previous methods and helps improve tasks like smart search assistants.
What this means in practice
- •For search engine developers: Enhance search systems by enabling indexes to autonomously self-improve based on user queries and simulated demands.
- •For automation platform teams: Improve efficiency and effectiveness of agents relying on document retrieval and historical interaction memory.
Authors
Sangam Lee, Wonjae Lee, Sunghwan Kim, Deogyong Kim, Jaehoon Kim, Daye Nam, SeongKu Kang, Dongha Lee
Abstract
Information retrieval is increasingly important as LLM agents tackle complex tasks involving diverse information needs. Because retrieval relies on an index that represents each document through index keys, retrieval quality depends heavily on how effectively these keys expose the knowledge contained in each document. However, effective index representations vary across retrieval environments, making it difficult for any fixed optimization strategy to perform consistently. Yet evolving an index to its retrieval environment remains largely human-driven, requiring humans to diagnose retrieval failures, refine the optimization strategy, and reprocess the index accordingly. We propose SELF-INDEX, a framework that enables an index to self-evolve without human intervention. Its Optimizer autonomously diagnoses retrieval shortfalls, selectively revises the responsible index keys, and validates each revision before updating the index. Beyond reacting to observed retrieval demands, SELF-INDEX proactively explores additional demands through a Query Simulator, allowing the index to evolve beyond the queries already available for optimization. Across diverse corpora and retrievers, SELF-INDEX consistently improves retrieval performance while outperforming existing index optimization methods. We further show that these benefits extend to downstream applications, improving the effectiveness and efficiency of search agents and helping agent memory systems retrieve useful past interactions.