Model concept geometry reveals uncertainty and execution patterns during inference
Lingtai: What Concept Geometry Reveals--and Does Not Reveal--About LLM Inference
Computation and Language
Summary
It’s hard to see what a large language model is thinking as it writes, without interfering. The authors created Lingtai, a tool that watches the model’s step-by-step activity by comparing it to a set of named concepts without needing extra training. Their findings show this concept-based view links closely to how uncertain the model is about its predictions, and that the model’s internal signals follow distinct patterns depending on the task. However, the tool doesn’t reliably show when the model is definitely correct, so external checks are still needed.
What this means in practice
- •For machine learning engineers: Integrate Lingtai’s concept telemetry to monitor language model uncertainty during code and math problem generation.
- •For ai system operators: Use task-conditioned signals from concept geometry to better understand model inference behaviors without retraining or labels.
Authors
Jiangang Chen
Abstract
Observing what a large language model computes during autoregressive inference--online and without training probes--remains difficult. We introduce Lingtai, a training-free concept telemetry layer: at each generation step, residual states are projected onto a domain-specific bank of named concept anchors, constructed without labeled concept examples, outcome labels, gradient fitting, or activation-space optimization, producing a structured per-step concept-coordinate signal. Across code generation and grade-school mathematical reasoning, this signal exhibits a robust association with predictive uncertainty: the association survives problem-identity and token-position controls and is not attributable to a single token type, is not explained by a simple correct/incorrect mixture on GSM8K, and is not reproduced by matched random anchors; it is markedly weaker or direction-inconsistent in K-means and PCA projections. Two structures emerge: a recurring uncertainty-linked activity signal whose functional geometry is task-conditioned (distinct activity-entropy shapes on HumanEval, MBPP, and GSM8K), and an execution-specific trajectory identity with strong local inertia but weak re-instantiation invariance--under completion-only elastic alignment, corruption at k=32 (approximately a median quarter of the completion) on the matched re-execution subset still retrieves the archived episode at 62.0%, while a fresh execution retrieves it only 11.7-16.0% of the time. Finally, a matched audit finds no evidence that the scalar concept-activity signal used here supplies a stable correctness coordinate under the tested protocol; we therefore treat correctness as externally supplied. Telemetry adds 0.7-1.6% per-token decode overhead for the 161-anchor code implementation, with unchanged generated tokens.