Retrieval is Enough: Training-Free Interpretability with a Tool-Using Agent
Machine LearningArtificial Intelligence
Summary
The authors study different ways to understand what neural networks are doing inside, comparing cheap methods that don’t need training to more costly ones that do. They create a new method called HARP, where a language model uses a database of neuron activations and text to ask questions and test ideas without any extra training. HARP works better than some training-based methods for finding and using concepts in the network's activations, while also being cheaper and more adaptable. Their findings suggest that methods requiring training may not give insights beyond what is already in the data, pushing for better tests of interpretability tools.
Authors
Sriram Balasubramanian, Soheil Feizi
Abstract
Interpretability methods for neural network activations span a wide cost spectrum, from cheap, training-free techniques (such as linear probes, PCA, SVD) to more expensive training-based ones (such as SAEs and activation oracles). Training-based methods are typically more powerful, in part because they leverage large activation datasets during training. This raises a natural question - do they actually surface insights that go beyond what is recoverable from the training dataset itself? To address this, we equip an LLM agent with a vector database of activations paired with their textual contexts, along with tools for manipulating activations - projecting out directions in latent space, computing activation differences and averages. The agent iteratively queries the database, forms hypotheses from the retrieved samples, and validates them by constructing linear probes. We call this method HARP, for Hypothesis-driven Agentic Retrieval and Probing. Despite not involving any training, HARP outperforms both activation oracles and SAE-based agents on concept discovery, concept detection, model steering, and secret elicitation. The training-free design also makes HARP substantially cheaper and more flexible: new datasets can be indexed on demand whenever existing ones prove insufficient. More broadly, our results suggest that current training-based methods do not yet extract insights beyond their training data, and motivate benchmarks that explicitly require interpretability methods to demonstrate such insights. We release our code at https://github.com/SriramB-98/HARP