KAP: Bridging the Knowledge Selection-Runtime Consumption Gap in LLM Systems
2026-07-27 • Machine Learning
Machine Learning
AI summaryⓘ
The authors identify a problem where large language models (LLMs) ignore the rich, structured information given to them and instead treat all input tokens equally, which slows down processing. They propose a new method called Knowledge Access Planning (KAP) that organizes this structured knowledge into a special plan used during decoding, allowing the model to access only the important information it needs. Their implementation, GraphSpec, shows that this approach can keep answer quality high while greatly reducing the amount of data the model has to handle, especially for very long inputs. This method helps LLMs work faster without changing the model or how it was trained.
Large Language ModelsKnowledge SelectionKey-Value StatePrompt EngineeringRuntime ConsumptionKnowledge Access PlanningIntermediate RepresentationGraphSpecLong-Context QADecoding Latency
Authors
Shuo Wang, Fang Xi, Wenyuan Huang, Qing Wang, Junming Su
Abstract
Modern LLM systems increasingly rely on knowledge-selection processes that produce high-value structured priors, such as ranked evidence, graph topology, multimodal alignment, and confidence signals. Yet LLM serving remains fundamentally oblivious to this rich structure: once such signals are serialized into a prompt, the backend observes only a flat token sequence, forcing dense and uniform consumption of the full key-value (KV) state during decoding. We term this architectural mismatch the Knowledge Selection-Runtime Consumption (KSRC) gap: richer contexts enlarge the full-prompt KV footprint and decode-time memory traffic, increasing latency and degrading throughput even when reasoning depends on only a small fraction of the context. To bridge the gap, we propose Knowledge Access Planning (KAP), a paradigm-shifting execution abstraction that elevates structured knowledge priors from passive prompt-construction hints into first-class physical execution artifacts. KAP establishes a universal intermediate representation (IR)-the runtime access plan-which compiles structured knowledge signals to govern physical KV access without altering logical prompt semantics, model weights, or training procedures. Through this IR, KAP shifts LLM serving from token-aware context consumption to plan-driven, knowledge-aware runtime consumption. We instantiate KAP with GraphSpec, a compiler-executor realization connecting structured knowledge selection to an LLM serving backend. We derive a phase-boundary model for the positive-speedup regime of plan-guided execution. Across 4K-128K long-context QA workloads, GraphSpec maintains answer quality comparable to full-context decoding while decoupling physical KV consumption from prompt length, reducing proposal-time KV access to 5.5% of source KV state at 128K, and fundamentally shifting the scaling trajectory of long-context generation.