Clinical AI platform matches top scribes in note quality and flexibility
Symphony for Text Generation: Benchmarking Clinical Note Generation
Machine LearningArtificial IntelligenceComputation and Language
Summary
Doctors are using new AI tools to help write notes about patient visits, but it wasn’t clear how well these notes turn out. The authors created a new dataset with medical conversations from three languages and tested several AI note-taking tools, including one called Corti. They found that Corti’s system writes notes just as well or better than popular tools, and it can be tweaked for different medical tasks. The paper also offers a new method to fairly compare such AI note generators and shares the dataset for future use.
What this means in practice
- •For clinical ai developers: Build and improve AI note-writing software that can be tuned for different medical documentation needs using the provided dataset and evaluation framework.
- •For hospital it teams: Evaluate and select ambient clinical documentation systems based on comparative performance data and quality dimensions from this study.
Authors
Daniel Varab, Victor Petrén Bach Hansen, Asbjørn W. Helge, Kevin Pelgrims, Mathias Baltzersen, Adrian Young-San Roessler, Vanessa Klungtvedt, Maximilian Brand, Lasse Krogsbøll, Henrik Cullen, Lars Maaløe
Abstract
Ambient documentation systems are rapidly gaining adoption, yet their impact on clinical note quality remains poorly characterized. We introduce MedConv, a multilingual dataset of 300 clinical encounters in English, Danish, and German, and use it alongside the Ambient Clinical Intelligence benchmark (ACI-BENCH) to compare Corti, a clinical AI platform, with two leading, accessible ambient scribe software applications built on general-purpose AI. We present a controlled clinical evaluation framework that combines entailment metrics with LLM-judged pairwise comparisons across eight dimensions adopted from PDSQI-9. Results show that Corti's API-based text-generation infrastructure is on par with or outperforms leading commercial scribes. We further show that Corti's configurable API provides the flexibility necessary to fine-tune quality dimensions for specific documentation use cases. We present the evaluation methodology and release a dataset to support future reproducible comparison of ambient documentation systems.