Scalable Circuit Learning for Interpreting Large Language Models
2026-06-15 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors focus on understanding how parts of large language models (LLMs) work together to produce outputs. Since individual neurons can have mixed meanings, they use features from a sparse autoencoder for clearer understanding, but this is usually computationally expensive. They introduce CircuitLasso, a faster method that finds important circuits in these features without heavy computation. Their approach reveals meaningful connections inside the model and helps improve performance on tasks that require generalizing to new situations with less computational cost.
mechanistic interpretabilitylarge language modelsneuronspolysemanticsparse autoencodercircuit learningsparse linear regressionintervention-based methodsdomain generalization
Authors
Naiyu Yin, Dennis Wei, Tian Gao, Amit Dhurandhar, Karthikeyan Natesan Ramamurthy, Yue Yu
Abstract
A prominent research direction in mechanistic interpretability is learning sparse circuits over LLM components to reveal how they jointly produce model behavior. However, raw neurons are polysemantic, making learned circuits hard to interpret. Sparse autoencoder (SAE) features alleviate this, but their high dimensionality makes existing intervention-based circuit learning methods computationally prohibitive. We propose CircuitLasso, a scalable circuit-learning approach based on sparse linear regression. CircuitLasso recovers circuits whose structural accuracy matches that of state-of-the-art intervention-based methods on the benchmark data, at a fraction of the computational cost. For interpretability, CircuitLasso efficiently uncovers relationships among SAE features, showing how human-interpretable semantic features propagate through the model and influence its predictions. Finally, we validate the utility of our learned circuits by leveraging their insights to achieve comparable performance at substantially lower cost on a domain-generalization task.