Knowledge organization improves large language model access to information
Knowledge-as-Skill: A Structural Design for Autonomous Knowledge-Base Use by LLM Agents
Information Retrieval
Summary
Large language models (LLMs) often rely on external knowledge bases to answer questions, but those knowledge bases can be hard for the model to understand and explore. The authors propose a new way to organize knowledge called Knowledge-as-Skill that makes it easier for AI agents to find, navigate, and make use of information. They structure knowledge into layers with descriptions that help the model decide what to retrieve and when, improving how well the AI uses the knowledge. Their initial tests show better factual accuracy and context recall compared to previous methods, though some trade-offs remain.
What this means in practice
- •For enterprise software teams: Structure and tag company knowledge bases to help AI agents retrieve more accurate answers in customer support automation.
- •For technical documentation writers: Create discoverable and navigable documentation formats that improve AI-assisted content search and summarization.
Authors
Jiangxu Wu
Abstract
Retrieval-augmented generation (RAG) gives large language models (LLMs) access to external knowledge, but its conventional retrieve-concatenate-generate pipeline makes retrieval decisions on behalf of the model. As tool use and agent loops become more reliable, an agent can decide whether to retrieve, what to inspect, and when to stop. This shift exposes a new bottleneck: the agent may not know what a knowledge base contains. Traditional knowledge bases expose documents as anonymous text chunks with limited information about scope, purpose, provenance, or relations. We propose Knowledge-as-Skill, an organization scheme that makes a knowledge base discoverable, navigable, and self-descriptive. It has three layers: a discovery layer centered on SKILL.md; a navigation layer with one index.md per directory; and a knowledge layer containing documents with YAML frontmatter for topic, type, provenance, and lifecycle. The design follows the Open Knowledge Format (OKF) and the Skill protocol without modifying the agent framework. We also provide knowledge-as-skill, a pipeline for converting heterogeneous collections of PDFs, Word files, web exports, and notes into this structure. In a preliminary evaluation on the WixQA enterprise customer-support benchmark, our setup obtains 0.889 Factuality and 0.816 Context Recall, compared with reported Corpus2Skill values of 0.767 and 0.708. It obtains slightly lower Faithfulness, lower Context Precision, and more interaction turns. Because the models, prompts, and knowledge-package construction differ, these results are directional cross-work evidence rather than a controlled comparison.