Llms need smart history selection for better clinical reasoning
Can LLMs Reason Over Long Horizons? An Empirical Evaluation of Context Strategies for Longitudinal Clinical Reasoning
Computation and LanguageArtificial IntelligenceMachine Learning
Summary
Understanding a patient’s health over time means looking at lots of medical history. The authors tested different ways of giving long medical histories to language models and found that just giving them all the info isn’t always helpful. Instead, picking and organizing the right parts of a patient’s history helps these models answer questions better. However, even the best methods sometimes struggle when the questions are based on stuff not actually supported by the history.
What this means in practice
- •For healthcare ai developers: Improve medical AI tools by using selective history summarization to enhance diagnosis support over long patient records.
- •For legal case review teams: Use episodic and hybrid context strategies to better extract relevant evidence from lengthy case documents for legal reasoning.
Authors
Taye Akinrele, Noorbakhsh Amiri Golilarz, Subash Neupane, Sudip Mittal, Shahram Rahimi
Abstract
Longitudinal clinical reasoning requires large language models (LLMs) to identify and integrate relevant evidence distributed across extended patient histories. Although long-context models can process increasingly large amounts of information, providing more history does not necessarily make relevant evidence more accessible or improve reasoning. We compare five context strategies (Full, Recent, Episodic, Semantic, and Hybrid) on MedLoCoMo across four open-weight LLMs, examining answer correctness, robustness to query-evidence distance, and abstention on questions with unsupported premises. Episodic and Hybrid generally achieve the strongest overall accuracy, while Recent Context degrades most as supporting evidence becomes more distant; Episodic and Hybrid maintain the highest accuracy at long distances. Analysis of adversarial questions further shows that strong performance on answerable questions does not necessarily translate to successful abstention when the available history does not support the requested conclusion. These findings show that reliable longitudinal reasoning depends not only on how much history an LLM can access, but critically on how relevant evidence is selected and presented for reasoning.