Large language models boost accuracy of stroke medical chart extraction
Large Language Models for Accurate Medical Chart Abstraction: Enabling Scalable and Secure Artificial Intelligence Deployment in Stroke
Computation and Language
Summary
Medical records often have important stroke information written in hard-to-use ways. The authors tested many advanced language AI models to pull out specific stroke details from thousands of medical reports. They found that one model, LLaMA3.3-70B, was very accurate, even more so than non-expert humans for most tasks. This approach can help hospitals organize stroke data better and faster, but experts still need to check tricky parts.
What this means in practice
- •For hospital data teams: Automatically extract detailed stroke intervention data from clinical reports to improve retrospective registries with expert-reviewed quality control.
- •For clinical informatics developers: Deploy locally run large language models that efficiently structure unorganized stroke intervention narratives for research and quality monitoring.
Authors
Zhusi Zhong, Carl M. Porto, David Hong, Miranda Gonzales, Revyn Kim, Gnaneswari Karayi, Lulu Bi, Joshua R. Feler, Elias Shaaya, Scott Collins, Liqi Shu, Grayson Baird, Mahesh Jayaraman, Shadi Yaghi, Zhicheng Jiao, Dylan N. Wolman
Abstract
Medical chart abstraction transforms unstructured clinical narratives into structured variables for research and quality monitoring. We evaluated a prompting-based large language model (LLM) framework for extracting eight clinical variables from 2,416 neurointerventional acute stroke intervention reports across three hospitals. Twenty-two instruction-tuned open-source LLMs were evaluated across architectures, model sizes, and biomedical adaptations using Quick Response and Chain-of-Thought (CoT) prompting. Non-expert staff annotations served as the cross-model reference standard. Evaluation was restricted to report-variable pairs with a valid LLaMA3.3-70B CoT prediction and a corresponding reference label. Performance measures included extraction accuracy, inference latency, and agreement with expert adjudication. Twenty discordant cases per variable underwent review by two senior physicians and four medical students. LLaMA3.3-70B achieved the highest overall accuracy (94.8%). CoT improved performance on inference-dependent variables such as site of occlusion, while Quick Response was optimal for directly stated procedural fields. Expert adjudication showed that AI predictions were more closely aligned with expert interpretations than non-expert annotations for most variables, with intracranial angioplasty remaining a challenge. Locally deployed LLMs can support scalable clinical information extraction and retrospective stroke registry curation, with expert oversight for ambiguous cases.