Large language models extract treatment effects in Rett syndrome clinical notes
Documenting Trofinetide Treatment Phenotypes in Rett Syndrome with a Dual-LLM Consensus Pipeline
Computation and Language
Summary
Rett syndrome is a rare disorder treated by trofinetide, but tracking how patients respond in real-world settings is hard because doctors' notes are long and complicated. The authors developed a computer method using two large language models that work together to read medical notes and spot specific treatment effects and symptoms automatically. This method was much more accurate than simple keyword searches and agreed well with expert opinions. It helps doctors and researchers understand patient responses over time without manual chart review.
What this means in practice
- •For hospital data teams: Automatically monitor treatment response for rare diseases like Rett syndrome by extracting phenotypes from clinical notes using LLMs consensus methods.
- •For medical software developers: Integrate a dual-LLM consensus pipeline into electronic medical record systems to improve detection of drug effects beyond keyword searches.$Commercial implications: Enables development of scalable medical record analytics software that offers superior real-world treatment monitoring for healthcare providers.
Authors
Daniel Palacios, Guantong Qi, Mikael Guzman Karlsson, Shawntel K. David, Mohammed Sameeruddin, Bernhard Suter, Zhandong Liu, Davut Pehlivan, Hu Chen
Abstract
Objective: Trofinetide, approved in 2023 as the first Rett syndrome treatment, lacks scalable real-world monitoring because chart review is labor-intensive. We developed a dual large language model (LLM) consensus pipeline to extract 18 treatment-response phenotypes, comprising 4 adverse effects, 13 Rett clinical features, and medication refusal, from clinical notes. Materials and Methods: We analyzed 26,763 notes from 108 patients with confirmed sustained trofinetide exposure at Texas Children's Hospital. The pipeline filters notes by keyword, classifies each through two LLMs requiring consensus, and aggregates a patient-by-timepoint matrix. Architecture and prompt were selected on a diarrhea development set, then frozen and applied to the other 17 phenotypes with the phenotype definition substituted, as an out-of-sample test. The reported prompt comparison scores 183 expert-labeled cells. Results: A prompt requiring diarrhea to be active and severe performed best (84.2 $\pm$ 0.0% accuracy; F1 = 0.674), 47 percentage points above regular-expression matching ($p < 0.0001$). Expert reviewers judged six of the pipeline's calls per phenotype, three positive and three negative, and agreed with 90.2% (92/102). The 13 Rett clinical features are manifestations of the disorder itself rather than adverse effects of trofinetide. Gross and fine motor impairment, language/communication, and stereotypy were documented present in 92-100% of patients at every visit. Discussion: In-context LLM extraction captured multi-domain treatment responses. The outcome is documented presence at each visit, not a direction-of-change judgment. Conclusion: Coupling in-context LLM reasoning with a consensus architecture offers a scalable alternative to keyword and regular-expression extraction for real-world evidence, adapting to new phenotypes and medications through configuration alone.