Hallucination neurons are detectable but not uniquely localized in large language models
Hallucination Neurons and Where to Find Them: An Investigation into the existence of Hallucination Neurons
Artificial Intelligence
Summary
Large language models sometimes make things up, called hallucinations, and some neurons in these models seem linked to this behavior. The authors tested if small groups of neurons truly control hallucinations or if many neurons share this role. They found that while it’s possible to identify neurons tied to hallucinations, these neurons aren’t unique and are strongly correlated with others. This means the way hallucination works inside the model is spread across many neurons, not just a few special ones.
What this means in practice
- •For model auditing teams: Improve methods to detect neuron groups linked to hallucination but avoid overconfidence about their unique role in model behavior.
- •For ai safety engineers: Use the proposed diagnostic protocol to validate neuron-level interventions when trying to reduce hallucination in language models.
Authors
Huseyin Cavus, Sebin Sabu, Joshua Spear, Jaskaran Singh Kawatra, Pavithra Rajendran
Abstract
Interpretable machine learning for Large Language Models (LLMs) increasingly relies on sparse probing methods that identify small sets of neurons claimed to detect and causally influence behaviors such as factuality recall, safety alignment, and hallucination. These claims have important implications for model auditing and behavioral steering, yet they are rarely tested against known failure modes of $L_1$-regularized probing in correlated, high-dimensional feature spaces. We propose a five-step diagnostic protocol covering feature correlation, bootstrap stability, sparse versus dense ranking disagreement, intervention baselines, and cross-dataset evaluation as a minimum standard for sparse-neuron localization claims. We investigate prior work using our proposed approach, specifically on H-neurons using open-source LLMs across TriviaQA, BioASQ, and NQ-Open datasets. Our results demonstrate detection replicates across both models and datasets, and exceeds the original reported AUROC gaps for TriviaQA and BioASQ datasets. Gemma 3 4B consistently outperforms MedGemma 4B on matched datasets, with AUROC gaps of +0.311 versus +0.235 on TriviaQA, +0.474 versus +0.455 on BioASQ, and +0.128 versus +0.112 on NQ-Open respectively. Causal validation at $n = 500$ with five random seeds shows statistically significant effects beyond random same-layer baselines. At the same time, the diagnostic results indicate that the selected neurons are not uniquely localized. Across the three Gemma 3 4B settings, 19 of 22 selected H-Neurons have Pearson $|r| > 0.7$ with other features, bootstrap selections show only moderate stability, and sparse and dense rankings overlap only weakly. Our findings show that sparse predictive structure can coexist with non-unique neuron selection. Routine diagnostic validation is necessary to distinguish detection claims from localization claims in mechanistic interpretability.