DocTalk improves medical question answering with fine-tuned language models
DocTalk: Enhancing Medical Knowledge Retrieval via Fine-tuned Large Language Models with Chain-of-Thought Reasoning for Robust Clinical Decision Support
Artificial IntelligenceComputation and LanguageComputer Vision and Pattern Recognition
Summary
Getting accurate medical answers from large AI language models is hard because they can make things up or misunderstand medical details. The authors created DocTalk, a system that uses expert-reviewed questions and patient discharge summaries to train models to give better, grounded answers. They fine-tuned several open-source models, which improved their ability to provide accurate and clinically relevant responses. DocTalk was also tested outside its own dataset and showed improved results, and it is now available as a secure web tool for doctors to query electronic health records.
What this means in practice
- •For hospital data teams: Deploy interactive systems for doctors to safely query patient records with grounded AI-generated summaries.
- •For healthcare ai developers: Improve medical question answering models by fine-tuning with clinician-designed, chain-of-thought QA pairs and instruction tuning.
Authors
Asad Gohar, Md Ariful Islam Mozumder, Moon-Il Joo, Hee-Cheol Kim
Abstract
Objective: The rapid advancement of large language models (LLMs) in healthcare has opened new possibilities for AI-assisted clinical decision-making; however, safe and effective deployment remains challenging because of hallucination and domain misalignment. This study introduces DocTalk, a doctor-assistive question-answering (QA) system designed to generate concise, chart-grounded responses from de-identified discharge summaries. Methods: We developed a high-fidelity data generation pipeline that produces 500 clinically relevant QA pairs through sentence-level decomposition, chain-of-thought-guided question formulation, and expert review. Six open-source LLMs, including three general-purpose and three biomedical models, were evaluated before and after parameter-efficient instruction tuning using Low-Rank Adaptation (LoRA). The framework integrates prompt engineering, in-context learning, and medically aligned evaluation to support factual and interpretable responses. Results: On DocTalk-QA, instruction tuning produced consistent improvements across lexical, semantic, and clinical fidelity metrics. The best-performing tuned model, Llama-3-Med42-8B, achieved ROUGE-L = 0.40, BERTScore-F1 = 0.76, and UMLS-F1 = 0.49. To examine external generalization beyond the DocTalk-QA dataset, the two best-performing models, Llama-3.1-8B-Instruct and Llama-3-Med42-8B, were evaluated on PubMedQA before and after instruction tuning. Accuracy improved from 0.931 to 0.977 for Llama-3.1-8B-Instruct and from 0.935 to 0.943 for Llama-3-Med42-8B. Conclusion: DocTalk was deployed as a secure web-based platform for interactive EHR querying. These findings suggest that clinically grounded instruction tuning can improve medical QA performance and support practical clinical decision support.