Language models need stricter tests to prove explanation skills
What Would Falsify It? A Variable Specific Evidence Standard for Mechanistic Claims About Self Explanation
Artificial IntelligenceMachine Learning
Summary
Sometimes when AI language models explain their answers, it's unclear if they truly 'understand' their reasoning or if they just make up a story afterwards. The authors suggest a new way to test these explanations by comparing them to special controls that keep everything the same except the key part being tested. Using this method, they found that models often fail these strict tests, meaning we can't yet be sure their explanations truly reflect how they produced the answers. This work calls for better proof that explanations are truly linked to the model's thought process.
What this means in practice
- •For machine learning engineers: Use variable-specific controls to verify if language model explanations truly reflect underlying computation.
- •For ai safety teams: Implement stricter evidence standards to identify when model explanations may lack causal relevance.
Authors
Arshia Eftekhari zadeh
Abstract
When a language model explains an answer it has already given, does it reuse the computation that produced the answer or reconstruct a story from the answer alone? Attribution, transportability and recoverability are each compatible with causal use without establishing it. We propose an evidence standard: pair each positive statistic with a variable specific null that removes the tested variable's identity while matching relevant nuisance dimensions as far as possible, and audit unmatched dimensions. We apply this standard to a known cause. A cue naming a wrong option raises the rate of choosing that option by 64 to 68 percentage points across three models. Explanations mention the cue in 1.8 percent of items or fewer in three of four models tested. Three estimator classes yield favorable statistics, but none establishes causal sensitivity to the cue contrast under its own control in the three-model analysis. In the strongest case, a recovered cue direction reaches $R^2$ of 0.95 and exceeds a geometry matched random direction in all three seeds, while a direction fitted by the same pipeline with cue labels scrambled reproduces 61 to 76 percent of its effect at comparable realized edit magnitude. A fourth model passes one interchange endpoint, but unequal edit magnitudes and a contrast that changes both cue identity and cue-answer agreement limit its interpretation. These experiments leave causal access unresolved. They establish an evidentiary requirement: favorable mechanistic statistics must survive controls for variable identity and nuisance structure. Reusable controls separate generic from identity specific transport effects, fit null directions with scrambled labels, and audit realized intervention magnitudes.