Progressive Disclosure for LLM-Maintained Wiki Knowledge Bases: a Preregistered Ablation
2026-07-06 • Computation and Language
Computation and LanguageComputers and SocietyInformation Retrieval
AI summaryⓘ
The authors studied how language model (LLM) agents answer questions using a large 709-page wiki by changing how the agent accesses the information. They expected that showing just a summary and only loading needed pages would reduce costs compared to loading a full index, but found that skilled agents often skip the index anyway. Instead, savings came from the agent accessing fewer pages and making fewer steps, without hurting answer quality. Their results show targeted page access improves efficiency more than just trying to avoid loading an index.
LLM agentsknowledge basesprogressive disclosuremarkdown wikiindexingretrievalself-routinganswer qualitytool-using agentsevaluation validity
Authors
Theodore O. Cochran
Abstract
LLM agents increasingly answer questions against knowledge bases they help maintain. A common intuition holds that progressive disclosure, a compact catalog plus a one-line summary per page so the agent loads only what it needs, should make this cheaper than consulting a large monolithic index. We test that on a real 709-page markdown wiki maintained by an LLM. We retrofit it for progressive disclosure and run a preregistered ablation in which four versions of the corpus differ only in how the agent reaches the content: page bodies are byte-identical across arms, frozen as immutable git tags, so any measured difference is due to access structure alone. We cross the arms with three access conditions (a protocol-constrained agent, a free self-routing agent, and a catalog-preload regime) and grade answers blind against verified gold references with a cross-family judge. A pilot upended the premise: a capable tool-using agent never loads the index, inferring a page's path from the question and reading it directly, so the specific saving the retrofit targets does not materialize. We therefore made answer quality primary and cost secondary. Quality is non-inferior (the retrieval arm matches the index baseline within the preregistered margin) while cost falls in every regime, from about a third for a self-routing agent to well over half under catalog-preload, all confidence intervals excluding zero. The saving comes not from avoiding the index load but from more targeted access: the retrieval arm cites fewer pages and takes fewer tool turns. The study doubles as a case study in evaluation validity, applying threat-to-validity discipline to the tooling that produced it.