Circuit-Diff helps pinpoint knowledge in AI model features and circuits
Circuit-Diff: Factual Edit-based Intervention Method for Localizing Knowledge in Attribution Graphs
Machine LearningArtificial Intelligence
Summary
AI models have many hidden parts that work together, but it’s hard to tell which parts hold what knowledge. This paper introduces Circuit-Diff, a way to edit an AI’s knowledge slightly and watch what parts of the model respond. The parts that change help reveal the model’s internal understanding related to that knowledge. This method helps make complex AI systems more understandable and testable.
What this means in practice
- •For ai model developers: Identify and verify specific knowledge encoded in model components to improve interpretability and debugging.
- •For software engineers: Use factual edits to target and modify relevant model features, enabling precise updates without full retraining.
Authors
Edward G. Friedman, Xiangchen Song
Abstract
Mechanistic interpretability defines features as the fundamental units of a neural network and circuits as the weighted subgraphs that carry out its computation. Because individual neurons are polysemantic, Cross-Layer Transcoders (CLTs) were introduced as a way to approximate a model's circuits by generating an attribution graph. The nodes of that graph, however, are unlabeled features: reading a graph means pruning it and then working out by hand what each surviving node means. To make CLTs easier to use for circuit discovery, we introduce Circuit-Diff, which intervenes on the model itself with a low-rank factual edit and takes the features whose role in the attribution graph changes under that edit as related to the edited knowledge. On the edits we examine, the flagged nodes are not only detectors of the object token: read off the CLT's released feature dashboards, they include features for the history, geography and associations surrounding the old and new objects. We formalize the method, measure how reliable a frozen CLT remains after a factual edit, test the selected nodes causally by patching them on up to 24 CounterFact edits, give a case study, and release an open-source implementation built on the circuit-tracer package, together with two further tools (multi-prompt aggregation and rule-based supernode labeling).