Summary
Sometimes, when AI models explain their thinking step by step, their explanations sound believable but don't truly match how the AI actually made its decision. The authors study whether the ideas mentioned during these explanations are the same as the important ideas the model used behind the scenes to get the answer. They find that while many important concepts are shared, those that really affect the answer show up most strongly in the middle layers of the model—not always at the end, and not all are spoken aloud. This means just looking at explanations or which pieces of information match isn’t enough to know if the explanation is trustworthy.
What this means in practice
- •For natural language processing engineers: Improve diagnostic tools to verify if step-by-step AI explanations truly drive final model answers by focusing on intermediate model layers.
- •For ai safety teams: Develop more reliable tests of AI reasoning faithfulness by identifying causal contributions of internal concepts to outputs.
Authors
Qianli Wang, Yilong Wang, Dennis Wei, Jingyi Sun, Simon Ostermann, Pepa Atanasova, Nils Feldhus
Abstract
Chain-of-thought (CoT) can sound plausible yet be unfaithful to the model's underlying reasoning. Most prior work probes CoT faithfulness through input--output behavior or input attributions, leaving internal computation largely underexplored. We instead cast faithfulness as internal concept grounding: Does a large language model's (LLM) CoT reasoning engage the same internal concepts that support the LLM's direct prediction, and do the shared concepts causally drive its answer? Encoding a prediction pass and a CoT pass with a single shared sparse autoencoder (SAE), a reliable approximator of the latent concepts LLMs use, makes their internal concepts directly comparable. We introduce three correlational metrics of concept-level alignment and a causal metric, $Δp$, which ablates the shared concepts and measures the drop in answer probability. Across five LLMs and four datasets, concept alignment is generally high, as indicated by the correlational metrics; yet these only identify which concepts are shared, not how much they causally contribute. $Δp$ fills this gap: causal faithfulness varies substantially with model depth, peaking at mid-to-late layers rather than the final ones, and model scale reshapes the layer-wise profile. Moreover, causally important shared concepts are not always verbalized in the CoT. These dissociations suggest that faithfulness cannot be reliably assessed from surface-level or representational correspondence alone; assessing it requires causal tests of whether the internal concepts underlying a CoT actually drive the model's prediction.