AI summaryⓘ
The authors studied how a method called LoRA fine-tuning helps large language models (LLMs) improve at ranking tasks, which means deciding what information is most relevant. They found that changing attention mechanisms mainly in the middle layers of the model is enough to get most of the ranking improvements, and skipping those changes hurts performance more than skipping changes elsewhere. They also showed that parts of the model where LoRA made big differences matched places where the model paid more attention to important information like rare words or matching between queries and documents. Overall, their work reveals how specific attention patterns develop during fine-tuning and suggests better ways to adapt ranking models.
LoRA fine-tuningLarge Language Models (LLMs)Attention mechanismsMLP (Multilayer Perceptron)RankLLaMARerankingLexical matchingRarity sensitivityQuery-document interactionInterpretable attention
Authors
Matthew Perlman, Atharva Nijasure, James Allan
Abstract
LoRA fine-tuning is standard for adapting LLMs to reranking, but it remains unclear where in the network task-specific relevance behavior is learned and what attention-level changes accompany that learning. Through ablation and attention experiments, we identify where LoRA attention updates to RankLLaMA improve performance and whether those gains coincide with interpretable relevance-oriented attention patterns such as lexical matching, rarity sensitivity, and query-document interaction. We find that given LoRA fine-tuned MLPs throughout the network, restricting LoRA attention updates to a compact mid-network region is sufficient for recovering over half of the performance gained by applying LoRA to all attention layers, and that omitting attention fine-tuning in this region hurts performance more than elsewhere in the network. Additionally, we show that regions where applying LoRA affects performance the most overlap with regions where fine-tuning increased attention to axiomatic IR features. Rarity sensitivity, document-query interaction, and several compositional features are highly correlated with gains in ranking performance. Our results support an interpretable, correlational account of how relevance-oriented behavior emerges during LoRA fine-tuning and point toward improved strategies for adapting rerankers.