Local sparsity improves unsupervised safety detection in large language models
Local Sparsity Enables Unsupervised LLM Safety Detection
Machine LearningArtificial Intelligence
Summary
Checking if AI language models say unsafe things is hard because new problems keep appearing that weren’t seen before. The authors find that looking only at what’s normal or safe can help spot unsafe inputs without needing examples of unsafe cases. They do this by focusing on small groups of active parts inside the AI that tend to work together, making it easier to see when something unusual happens. Their approach works well even when given very little unsafe data to adjust the detection.
What this means in practice
- •For ai safety teams: Detect new and unforeseen unsafe inputs in large language models without relying on labeled harmful examples.
- •For chatbot developers: Improve runtime detection of unsafe user inputs by efficiently focusing computation on a small subset of neural activations.
Authors
Xin Chen, Gil Kur, Alexander Shevchenko, Andreas Krause
Abstract
Deployment-time safety methods for large language models (LLMs) are predominantly supervised and assume access to unsafe training data. Nevertheless, new attacks and harm categories regularly arise, not captured by models trained in such a supervised fashion. An alternative approach is to view this problem through the lens of anomaly detection, namely, to rely solely on modeling safe data and flagging out-of-distribution inputs. However, LLM activations lie in a high-dimensional space, raising concerns about whether anomaly detection is statistically feasible. We show that, under the linear representation hypothesis (LRH), there may indeed be hope. In the LRH concept space, which is typically recovered via a sparse autoencoder (SAE), nearby points share a small common active support. Using this local sparsity insight, we propose a framework for locally masked SAE-based anomaly detection, supported by theoretical justifications. We validate it on various architectures and datasets, including both capability-testing datasets and safety-specific datasets. Finally, when we allow algorithms to use 1% out-of-distribution data for calibration, locally sparse methods achieve near-optimal performance, demonstrating their ability to capture meaningful safety information while using only 1-2% of SAE neurons for computation.