SDoH-Aware Narrative Anchoring Bias in Medical LLMs for Trustworthy Clinical Decision Support
2026-08-24 • Computation and Language
Computation and LanguageArtificial Intelligence
AI summaryⓘ
The authors studied how medical language models answer questions about patients when the same medical case is told in different ways (patient voices). They found that even if the model knows the right answer, it might change its response based on how the story is told. They tested three versions of a model called Qwen2.5 and saw that the biggest one was most accurate but still made about 32% inconsistent errors. The authors suggest that it’s important to check not just if these models give correct answers, but also if their answers stay consistent across different but equivalent patient stories.
Medical large language modelsSocial Determinants of Health (SDoH)Narrative anchoring biasCounterfactual datasetInstruction tuned LLMsQwen2.5 modelsClinical decision supportModel accuracyConsistency in NLPMcNemar exact test
Authors
Ahnaf Atef Choudhury, Ramkrishna Saha
Abstract
Medical large language models are often judged by how many clinical questions they answer correctly. That view is useful, but it misses a practical risk. A model may know the right answer and still change its response when the same case is written in a different patient voice. This paper evaluates that risk as SDoH aware narrative anchoring bias. We use NarrativeShield SDoH MedQA, a counterfactual medical question answering dataset in which each case appears in persona based narratives while the answer key remains fixed. The dataset is reshaped from wide format into case grouped persona rows. We evaluate three open source instruction tuned LLMs from the Qwen2.5 family: 1.5B, 3B, and 7B. The final experiment uses 300 clinical cases and produces 8,100 model responses across three prompting conditions. We report persona level accuracy, counterfactual consistency, correct consistency, and narrative sensitivity error. Qwen2.5 7B achieves the best accuracy at 56.33 percent and the best correct consistency at 40.33 percent. Paired McNemar exact tests show significant accuracy gains for 7B over 3B in all prompt settings. Even so, narrative sensitivity remains, with the lowest error still at 31.67 percent. These results suggest that trustworthy clinical decision support should be evaluated by both average correctness and stability across medically equivalent patient narratives.