Fine-tuning changes in large language model workings do not predict task importance
Decoupling Internal Representational Changes and Causal Importance in Fine-Tuned Large Language Models
Artificial Intelligence
Summary
Fine-tuning large language models helps them perform new tasks, but how it changes their inner workings is unclear. The authors studied how fine-tuning affects parts of the models and found that the important components for tasks are usually in specific layers. However, these important parts don’t match up with the layers that change the most during fine-tuning. Also, even if two different tasks share some important parts, improving one task can sometimes hurt the other, especially when their goals are very different.
What this means in practice
- •For machine learning engineers: Identify which model layers truly drive specific task performance to guide more targeted fine-tuning strategies.
- •For natural language processing developers: Avoid unintended performance drops when fine-tuning on multiple different types of tasks by understanding component overlaps.
Authors
Lingfang Li, Procheta Sen, Shubham Das, Danushka Bollegala
Abstract
Fine-tuning has emerged as a widely adopted approach for adapting LLMs to a variety of downstream tasks. However, how it reshapes their internal mechanisms remains poorly understood. To address this, we investigate how fine-tuning alters internal representations in LLMs, including attention patterns and layer-wise activations, and examine whether these changes are linked to task-relevant components identified by EAP (e.g., attention heads and logit-level activations) that drive task performance. We find that EAP-identified components are concentrated within specific layers, indicating a degree of functional localisation in how models internalise task-specific behavior. Notably, the distribution of these components across layers is largely uncorrelated with the layers undergoing the most substantial representational changes during fine-tuning. Furthermore, we observe that overlap in EAP-identified components across tasks does not translate into cross-task performance transfer if the tasks are different in nature (e.g. classification vs. generative tasks). More specifically, fine-tuning on one task can lead to a degradation of performance on another when the two tasks exhibit a high degree of overlap in their EAP-identified components.