Knowledge Knows, Verbalization Tells: Disentangling Latent Directions for Mathematical Solvability in LLMs
2026-07-06 • Computation and Language
Computation and LanguageMachine Learning
AI summaryⓘ
The authors studied how large language models (LLMs) understand whether math problems can be solved. They found that the models separately store knowledge about solvability and the way they explain it (verbalization) in their internal workings. Changes in the models' answers that seem made up are mostly due to shifts in verbalization, not actual knowledge. By using specific prompts or techniques, the authors showed it is possible to adjust these internal parts to reduce false answers and improve when the model wisely chooses not to answer.
large language modelsmathematical reasoningsolvabilityverbalizationinternal representationslinear decodingmodel fabricationpromptingactivation steeringmodel abstention
Authors
Nikolaos Xiros, Maria-Eleni Zoumpoulidi, Georgios Paraskevopoulos
Abstract
Although LLMs have made significant progress in mathematical reasoning, determining whether a mathematical problem is solvable remains a fundamental yet challenging capability. While recent studies have probed internal representations of model solvability beliefs, verbalization has primarily been studied behaviorally rather than as an internal representation, limiting its analysis and manipulation. We address this gap by separately probing representations of solvability knowledge and verbalization, allowing us to disentangle the two within model hidden states. Across multiple LLMs, we show that knowledge and verbalization are encoded as distinct, linearly decodable representations and that fabrication is primarily associated with changes in verbalization rather than the underlying knowledge. Prompting with unsolvability cues reduces fabrication primarily by shifting verbalization, while activation steering demonstrates that these representations can be echanistically manipulated to improve model abstention.