Large language models confused by incidental patient talk in notes

Incidental information contaminates patient notes and disrupts clinical reasoning in large language models

Computation and Language

Summary

Large language models used to write and reason about patient notes can get confused by small talk or background conversations that are not directly related to the patient’s condition. The authors studied hundreds of doctor-patient dialogues and found that these models often include irrelevant chat in the notes and sometimes use that information incorrectly in medical reasoning. They also showed that speech from nearby patient encounters can leak into transcriptions and affect the notes generated by certain language models. The authors suggest checking how well models resist this distraction before using them in clinical settings and safeguarding note-taking without harming their reasoning abilities.

What this means in practice

  • For hospital data teams: Detect and reduce incidences of irrelevant speech leaking into clinical notes during automated transcription to improve documentation reliability.
  • For healthcare software developers: Design safeguards in clinical documentation tools to prevent contamination from unrelated conversations while preserving accurate clinical reasoning by language models.

Authors

Krithik Vishwanath, Brandon Ye, Anton Alyakin, John E. Markert, Aaron Hsieh, Michał Mańkowski, Eric K. Oermann

Abstract

Large language models (LLMs) are increasingly relied upon to support ambient documentation and clinical reasoning. Here we examine the impact of a failure mode shared between these two applications by assessing their sensitivity to information incidental to the patient encounter. In 576 patient-clinician dialogues, we found that frontier models inserted small-talk exchanges into 35% of notes, while mean quality scores changed by at most 0.20 points on five-point scales. In 3.7% of frontier notes, models misattributed the asides or used them clinically. In 57 mock recorded consultations, background speech from a separate patient encounter at -10 dB leaked into 48.2% of transcripts, with contamination detected in 5.3% of downstream notes generated by four open-weight models. We propose a dual encoding hypothesis of clinical reasoning and distraction in LLMs, with preliminary evidence that LLM components associated with disruption by incidental information also support clinical reasoning. These findings support evaluating resistance to incidental information before clinical use, with safeguards that prevent contamination while preserving clinical reasoning.
View PDFOpen arXiv