Experienceindex aids ai agents in reusing knowledge about artifacts
ExperienceIndex: Artifact-Grounded Memory
Computation and LanguageArtificial IntelligenceInformation RetrievalMachine Learning
Summary
When AI systems answer many questions based on large collections of documents, they often forget what they learned before, making their answers slower and less accurate. The authors created ExperienceIndex, which helps AI remember specific information about individual documents and how they relate to each other from previous tasks. This memory helps AI find all the right documents faster and give better answers. ExperienceIndex works across different types of tasks and AI models, and even helps weaker models improve by learning from stronger ones.
What this means in practice
- •For legal teams: Improve AI tools that quickly identify all relevant court cases needed for answering legal questions by using prior experience about case documents.
- •For data analytics teams: Enhance AI assistants that summarize scientific literature by reusing knowledge from previous queries about related research articles.
Authors
Peter Baile Chen, Geoffrey X. Yu, Xinming Liu, Samuel Madden, Dan Roth, Jacob Andreas, Doug Downey, Michael Cafarella
Abstract
Knowledge-intensive tasks require answering many questions by reasoning about a shared corpus of artifacts (e.g., court cases, or scientific literature). As humans interact with these corpora, they naturally accumulate experiential knowledge about artifacts, enabling them to quickly identify the complete set of relevant artifacts for each new task. However, existing AI agents lack appropriate memory solutions to build or reuse such artifact-grounded experience, leading to lower answer quality and higher online cost. Existing memory solutions extract and reuse information from prior task-solving traces, but they primarily focus on user preferences, factual attributes, or abstract reasoning patterns rather than persistent artifact-specific knowledge. We introduce ExperienceIndex, a novel experience layer for AI agents that captures and reuses knowledge about artifacts based on prior reasoning traces. ExperienceIndex stores two complementary forms of experience: (i) single-artifact experiences that summarize an artifact's contribution to prior tasks and (ii) artifact-pair experiences that encode structural relationships discovered during past reasoning. Integrated as lightweight middleware, ExperienceIndex uses an experience retrieval mechanism to guide agents toward the complete set of relevant artifacts for new tasks, improving both answer quality and efficiency. Across diverse corpora and agentic solutions with different search frameworks, ExperienceIndex delivers consistent gains, raising answer quality by up to 11.0 points and reducing online dollar cost by up to 50.5%. We further demonstrate two benefits: (i) cross-task generalization, where experiences accumulated from text-to-SQL tasks transfer to factoid QA tasks over the same artifact corpus, and (ii) teacher-student learning, where experiences from a stronger model enable a weaker model to reach comparable performance.