Precision steering improves knowledge conflict resolution in language models
Key Path Identification for Resolving Knowledge Conflicts via SAE-based Steering
Artificial Intelligence
Summary
Large language models sometimes get confused when given conflicting information. The authors found that instead of changing many parts of the model all at once, focusing on a few key parts that strongly influence others can work better. They created a new method that identifies these important features and changes only them, making the model more accurate and less noisy. Their tests showed this approach improved performance by a noticeable margin compared to older methods that changed many features at once.
What this means in practice
- •For machine learning engineers: Improve language model accuracy in applications where conflicting contextual knowledge affects outputs by targeting only key influencing features during model updates.
- •For ai product teams: Reduce unwanted side effects when updating LLMs with new knowledge by using precise feature steering to maintain fidelity to context.
Authors
Wenbo Zhang, Zhongxiang Sun, Zhiguang Han, Jun Xu
Abstract
Sparse autoencoder (SAE)-based steering has been widely used to address knowledge conflicts by guiding LLMs to be more faithful to the contextual knowledge. Existing methods usually perform mass steering, which modifies a large batch of SAE features identified via correlation-based methods. However, due to the inaccurate correlation and the neglected feature interactions, mass steering methods fail to precisely identify the features that play the key roles in steering and introduce a large number of redundant ones, which add noise and weaken the steering effects. Our empirical studies reveal that steering only a small subset of the identified features can achieve comparable or even better performance. Motivated by this finding, we propose Key Path Identification (KPI), a novel method that identifies key steering features characterized by strong causal dependencies with both upstream and downstream features. From these features, KPI constructs key paths and steers through less feature modifications. In this way, KPI advances SAE-based steering from quantity-driven to quality-focused, offering a perspective for more precise and interpretable model editing. Experiments in RAG tasks with knowledge conflicts show that our method improves the accuracy by 18% on average compared to the best baseline of mass steering, effectively filtering redundant features, alleviating side effects and demonstrating the core role of key paths in steering.