Feed-forward neurons signal hallucinations in large language models
NeuroActiSep: Detecting Factual Hallucinations from Feed-Forward Neurons in a Single Pass
Computation and LanguageArtificial IntelligenceMachine Learning
Summary
Language models sometimes make up facts, which makes them less reliable. The authors found specific neurons in these models that are connected to when the model hallucinates or invents information. They created a way to pick out these neurons and used them to build tools that detect when the model is likely making things up. Their method works as well as other techniques that look at more complex internal data in the model. They also studied how the position of neurons in the model affects how well hallucinations can be spotted.
What this means in practice
- •For ai developers: Create efficient tools to detect false answers in language models using identified neurons, improving trust without requiring complex analysis.
- •For natural language processing teams: Incorporate neuron-based hallucination detection to monitor and reduce factual errors in automated question answering systems.
Authors
Ali Derogar Odolou, Reza Nazari, Mostafa Salehi
Abstract
Hallucination in large language models reduces their reliability and slows adoption. Various white-box studies have used internal representations to detect patterns of truthfulness and factuality. A less-studied approach is to identify feed-forward neurons correlated with hallucination. We propose a method to rank feed-forward neurons at the final prompt token using a custom neuron selection dataset. We transfer the selected neuron identities to train hallucination classifiers on other factual question answering datasets. Our work provides empirical evidence that probes trained using the features from the selected neurons perform on par with probes trained on internal states. We also analyze the distribution of selected neurons and the effect of layer depth on detection performance.