Summary
Emergency departments check patients who return soon after their first visit to find ways to improve care. The study looked at doctors and an AI called GPT-4 to decide if follow-up reviews were needed based on diagnosis pairs. GPT-4 suggested follow-ups almost all the time, much more than doctors, possibly because it was not specially guided. The researchers built a tool using AI and a knowledge graph that better matched doctors’ judgments, helping flag important cases without adding too much review work. This work points toward smarter AI helping hospitals focus on the most concerning return visits.
What this means in practice
- •For hospital data teams: Integrate AI-enhanced knowledge graphs to automatically flag emergency revisit cases that likely need deeper review for quality improvement.
- •For healthcare ai developers: Design clinical decision-support tools combining large language models and knowledge graphs to better prioritize patient revisit screening.
Tested on one dataset.
Authors
Jonathan A. Handler, Marlene I. Robles-Granda, Jacob E. Mefford, Jeremy S. McGarvey, Gregory S. Podolej, Colleen J. Klein, Matthew D. Dalstrom, William F. Bond
Abstract
Background: Emergency Department (ED) return visits are commonly reviewed for quality assurance, but are often limited (e.g., to revisits within 48-72 hours) to increase actionable finding yield while minimizing chart review burden. Those limitations may lead to missed quality improvement opportunities. Methods: We conducted an exploratory, retrospective study of randomly selected ED visits to a multihospital health system having an ED revisit within 1-14 days to the same health system. Given only each visit's primary diagnosis, raters (2-3 clinicians and GPT-4 large language model [LLM]) assessed characteristics of the diagnosis pairs, including the "target": whether a pair warranted further assessment. Informed by rater response analyses, an algorithm leveraging an LLM-populated knowledge graph ("KGA") was created to automatically screen for potentially concerning pairs, then preliminarily assessed. Results: 99 diagnosis pairs were included. GPT-4 responses poorly correlated to clinician raters, rating nearly all (94%) pairs as warranting follow-up (4.4-13.3 times more than clinicians). However, prompt engineering was minimal. Among clinician raters, revisit medical gravity was consistently significantly associated with the target, while a differential diagnosis/complication composite was significantly associated on unadjusted, but not adjusted (though less powered) analysis. The KGA achieved 83-100% positive predictive value for at least one clinician rater determining further assessment was warranted based on the diagnosis pair. Conclusion: These results can inform next steps for improving screening with LLMs like ChatGPT. Further research is warranted to validate this preliminary work's finding that the KGA may enable enhancing the scope and yield of screening without substantially increasing reviewer workload.