LLM hidden states reveal needed capabilities before answering
Structured but Silent: Probing Capability Requirements in LLM Hidden States
Computation and Language
Summary
Some computer programs, called large language models (LLMs), can understand and answer questions. This paper shows that before these models give an answer, their internal 'hidden' data already contains clues about what kind of tools or skills are needed to respond. However, the models don’t always say this out loud when asked directly. The authors created a way to check if those hidden clues can be found easily, revealing that the models 'know' more than they express.
What this means in practice
- •For software developers: Detect required external capabilities from LLM internal states to improve automated tool selection before response generation.
- •For ai system integrators: Design systems that interpret LLM hidden data for better understanding of user intent and tool applicability even when models don’t verbalize it.
Authors
Kyojun Choo, Minsoo Song, Yunju Kang, Chanjun Park
Abstract
Reliable tool use requires more than triggering a mechanism or matching a query to an API description. Before selecting a specific tool, an agent must first infer the capability requirements implied by the user query. In this paper, we investigate whether these query-side capability requirements are linearly decodable from LLM hidden representations prior to generation, and how this hidden-state accessibility compares with explicit verbal classification. We introduce TACIT, a framework that decomposes external requirements along three fundamental axes: Source, Transformation, and World Effect, defining eight structurally distinct capability classes. Using 1,600 balanced training queries from benchmarks, synthetic examples, and new domain scenarios, we train linear probes on pre-generation hidden states from four open-weight LLM families. Our empirical results demonstrate that fine-grained capability structures are linearly decodable with high accuracy across all models. Crucially, however, we expose a representation-to-verbalization gap: these same models are significantly less reliable when asked to explicitly classify the same queries in natural language. This disconnect indicates that information about required external capabilities is linearly accessible in LLM hidden representations but not reliably expressed, a phenomenon we define as "structured but silent."