Probe weights do not reveal model causal features in truth detection

Decodability is Not Causality: Dissociating Probe Readouts from Behavioral Drivers via SAE Decomposition

Artificial Intelligence

Summary

This paper shows that just because you can use a linear probe to read information from a language model’s inner signals doesn’t mean those signals actually cause the model’s behavior. The authors explain that features aligned with the probe’s weights can be different from the ones the model really uses to decide if something is true or false. They created a new method combining probes with feature activity and intervention techniques to find which features truly affect behavior. Their tests on different datasets and models confirm that combining these approaches reveals more about what actually drives model decisions than the probe alone.

What this means in practice

  • For ai model developers: Identify which internal model features actually influence truth-related outputs to improve model interpretability and debugging.
  • For nlp tool builders: Refine truthfulness or deception detectors by focusing on causally relevant internal representations rather than only probe readouts.

Authors

Devesh Tiwari, Camille Davis, Shivank Sinha, Talia Weaver, Aditya Shah, Maheep Chaudhary

Abstract

Linear probes can decode safety-relevant concepts such as truthfulness from language-model activations, but probe accuracy may show only decodability, not that the features the probe weights causally drive model behavior. We demonstrate that this gap cannot be closed from the geometry of probe weights alone: the features geometrically aligned with probe direction need not be the ones the model uses, so causal relevance requires intervention. We introduce a feature-level diagnostic that decomposes a deployed True/False probe into sparse-autoencoder (SAE) features, ranks those features by both probe alignment and by gradient sensitivity of the model's behavior, and ablates the resulting shared, probe-only, and random feature sets under a coherence gate. On the truth probe of Buerger et al. (2024) (TTPD), applied in the instructed truth/deception setting of Long et al. (2025) for Gemma2-9B-Instruct, the two rankings overlap only weakly (about 12%, Spearman rho = 0.10), and ablation dissociates them sharply: features the probe shares with the model flip the output far more (up to 27%) than equally sized probe-only (6%) or random (1%) features at full coherence, while probe-only features instead perturb the probe's own readout. The dissociation holds across five seeds and a held-out split, and an activation-aware selection of features flips behavior nearly three times as often as the probe's geometric top features (17.6% vs. 6.1%). In this setting, therefore, the geometric projection of a probe's weight vector alone does not identify the features the model causally uses; however, combining probe information with feature activation statistics recovers substantially more behaviorally causal features, and coherence-gated SAE intervention is needed to separate them from probe readouts.
View PDFOpen arXiv