Key features improve steering of models with less noise and better accuracy
Key Path Identification for Resolving Knowledge Conflicts via SAE-based Steering
Artificial Intelligence
Summary
Large language models sometimes receive conflicting information that makes them give wrong answers. The authors found that changing only a few important features inside the model can fix these problems better than changing many features at once. They developed a method called Key Path Identification that picks out the most important features connected in a meaningful way, allowing for smarter and clearer adjustments. This method showed an 18% improvement in accuracy on tasks where knowledge conflicts happen. It also reduces unwanted side effects by avoiding unnecessary changes.
large language modelsknowledge conflictssparse autoencoderfeature steeringfeature interactionscausal dependencieskey path identificationmodel editingretrieval-augmented generationmass steering
Authors
Wenbo Zhang, Zhongxiang Sun, Zhiguang Han, Jun Xu
Abstract
Sparse autoencoder (SAE)-based steering has been widely used to address knowledge conflicts by guiding LLMs to be more faithful to the contextual knowledge. Existing methods usually perform mass steering, which modifies a large batch of SAE features identified via correlation-based methods. However, due to the inaccurate correlation and the neglected feature interactions, mass steering methods fail to precisely identify the features that play the key roles in steering and introduce a large number of redundant ones, which add noise and weaken the steering effects. Our empirical studies reveal that steering only a small subset of the identified features can achieve comparable or even better performance. Motivated by this finding, we propose Key Path Identification (KPI), a novel method that identifies key steering features characterized by strong causal dependencies with both upstream and downstream features. From these features, KPI constructs key paths and steers through less feature modifications. In this way, KPI advances SAE-based steering from quantity-driven to quality-focused, offering a perspective for more precise and interpretable model editing. Experiments in RAG tasks with knowledge conflicts show that our method improves the accuracy by 18% on average compared to the best baseline of mass steering, effectively filtering redundant features, alleviating side effects and demonstrating the core role of key paths in steering.