Resonant Context Anchoring: Decoupling Attention Routing and Signal Gain at Inference Time
2026-06-01 • Computation and Language
Computation and LanguageMachine Learning
AI summaryⓘ
The authors present a new method called Resonant Context Anchoring (RCA) to help large language models better use the evidence given to them, especially when it conflicts with what they 'remember' internally. Their approach adjusts how the model pays attention to input details without changing the attention probabilities, making the model more likely to stick to truthful information during responses. Tests on the Llama-3 models show that RCA reduces hallucinations and improves accuracy without needing extra training or slowing down the model. This makes RCA a simple and efficient fix to improve fact-based answers from language models.
Large Language ModelsContextual DisregardParametric MemorySelf-AttentionResidual StreamSignal-to-Noise RatioFactual HallucinationsInference-Time InterventionLlama-3Contrastive Decoding
Authors
Mingkuan Zhao, Yide Gao, Wentao Hu, Suquan Chen, Tianchen Huang, Zhenhua An, Zetao Chang, Xiayu Sun, Yuheng Min
Abstract
Large Language Models (LLMs) frequently exhibit "contextual disregard" when faced with input evidence that conflicts with their internal parametric memory, leading to persistent factual hallucinations. Existing mitigation strategies primarily rely on suppressing specific neuron activations or employing computationally expensive contrastive decoding mechanisms, which often result in increased perplexity or significantly elevated inference latency. To address these limitations, we propose Resonant Context Anchoring (RCA), a lightweight inference-time intervention method grounded in the perspective of residual stream signal dynamics. RCA aims to resolve the signal attenuation of external evidence during its propagation through deep networks. The core mechanism involves the orthogonal decoupling of routing logic and information magnitude within the self-attention module. By utilizing raw pre-softmax attention scores as an instantaneous metric of semantic alignment, we construct a dynamic gain field via non-linear rectification to selectively amplify the norms of value vectors corresponding to context tokens, without altering the attention probability distribution. This mechanism effectively elevates the signal-to-noise ratio (SNR) of input evidence within the residual stream mixture, thereby robustly anchoring the generation trajectory to the truthful context during inference. Extensive experiments on the Llama-3 model series demonstrate that RCA significantly improves contextual faithfulness across multiple factual consistency and strong knowledge-conflict tasks, effectively suppressing parametric hallucinations. Furthermore, results confirm that as a training-free and computationally negligible plug-and-play module, RCA achieves a Pareto improvement in faithfulness and fluency while maintaining the model's general language understanding capabilities.