General purpose large language model outperforms specialist medical model on jargon tasks
Domain-Specific Jargon in Large Language Models: A Comparative Analysis between General-Purpose and Specialist Models
Computation and LanguageArtificial IntelligenceMachine Learning
Summary
This paper looks at how well large language models understand tricky specialized words, especially in medicine. Surprisingly, a general model trained on many topics did better than one fine-tuned just on medical language. The researchers found that fine-tuning didn’t reorganize what the model knew but instead made it rely too much on a few parts of the model. Adjusting those parts brought performance closer to the general model. This suggests that tailoring a model for a particular field might not always improve its grasp of specialized jargon.
What this means in practice
- •For natural language processing engineers: Improve jargon comprehension by analyzing and adjusting key model components rather than relying only on domain-specific fine-tuning.
- •For technical content moderation teams: Develop better evaluation strategies for domain-specific language understanding using the new medical jargon benchmarks introduced.
Authors
Darin Keng, Zhewei Sun
Abstract
Large Language Models (LLMs) have shown remarkable proficiency on general-purpose tasks, yet their performance often degrades in highly-specialized technical domains. Moreover, little is known about how parametric knowledge of domain-specific terms is encoded within these models. We address this gap by contributing two novel medical jargon evaluation benchmarks and evaluate a general-purpose Llama-3.1 model against a variant fine-tuned on medical-domain data. Surprisingly, the general-purpose model outperforms the medically fine-tuned model on both tasks. Using mechanistic interpretability tools, we find systematic patterns of miscalibration for the medically fine-tuned model. Instead of reorganizing parametric knowledge, the fine-tuned model places greater emphasis on a small subset of model components associated with jargon-favoring predictions. We find that applying component reweighting strategies against the benchmark tasks successfully suppresses these components and closes the gap with the general-purpose baseline. We also observe that some jargon-sensitive components transfer knowledge to the same tasks involving materials science jargon, suggesting they encode a partially domain-agnostic notion of specialized terminology. Our results provide a case study in which a medically fine-tuned checkpoint does not improve jargon comprehension over its general-purpose counterpart, highlighting that domain adaptation should not be assumed to yield better performance on specialized terminology.