Clinical concepts found in language model brain areas improve health answers
Clinical Concept Centers in LLMs
Computation and LanguageMachine Learning
Summary
Large language models (LLMs) are used in healthcare, but usually we trust what they say without checking what they really 'think' inside. This paper shows that LLMs have special areas inside their model that hold medical ideas, and these areas actually drive their answers. The researchers found these 'concept centers' work well even when the model is tested in tricky scenarios, and making the model focus on them helps it give better medical advice. Doctors also preferred answers when these concept areas were active, suggesting these internal parts matter for clinical use.
What this means in practice
- •For clinical ai developers: Improve medical AI accuracy by directing models to use verified clinical concept centers in realistic healthcare settings.
- •For hospital data teams: Evaluate and monitor clinical language models internally by checking activation of concept centers to ensure reliable decision support.
Authors
Aishik Nagar, Abhishek Vaidyanathan, Arun-Kumar Kaliya-Perumal, Elijah Tzen Hsuen Boey, Stefan Winkler
Abstract
Large language models are increasingly used in clinical settings. However, research into the reliability and performance of these models has focused almost entirely on the language substrate, scoring what the model says. Mechanistic interpretability has found that the latent space carries a higher fidelity of representation than the text: internal representations not only encode substantially more than the output verbalizes, but the stated reasoning also systematically omits features that causally drive the answer. An evaluation of model behavior in terms of mechanistic interpretability has not been explored in clinical decision support. In this work, we extend behavioral evaluation into the latent space and ask whether clinical concepts exist as locatable, causally used representations inside open-weight LLMs. We find dedicated clinical concept centers in the latent space of all eleven open models we test. These concept centers are interpretable, firing only on their aligned clinical narratives, and meaningfully and causally drive model behavior in both constrained and open-ended settings. They are not just analytical representations, but circuits that can be utilized in clinical practice, and we explore their use from the perspective of both evaluation and performance. From the evaluation standpoint, models stay internally coherent and keep using the relevant concept centers even under adversarial role-based priming, while aligned priming improves downstream clinical performance. From a performance perspective, we simulate realistic deployment settings and find that steering models along these centers leads to meaningful downstream improvements. Finally, we conduct a blinded clinician validation and find the activation and usage of these concept centers predicts clinicians preferences.