End-to-end models improve automatic clinical note writing from long doctor talks
Bridging the Modality Gap in Long-Form Clinical Audio: A Comparative Study of Lightweight and Heavyweight End-to-End SOAP Generation
SoundArtificial Intelligence
Summary
It is hard for current computer systems to turn long doctor-patient audio recordings into accurate medical notes without errors or missing details. The authors made a new system that directly turns audio into structured clinical notes without needing to write out the spoken words first. They trained their system on a huge dataset and improved it step-by-step. Bigger versions of their model worked better, and their approach beat other common methods that use speech recognition followed by text processing.
What this means in practice
- •For healthcare it teams: Integrate end-to-end audio-to-note systems to reduce manual clinical documentation workload.
- •For medical transcription service providers: Develop more accurate automated note generation tools that bypass intermediate text transcripts for faster processing.$Commercial implications: Enables creating advanced transcription products that directly generate structured clinical notes from audio, improving speed and accuracy.
Authors
Ziyu Zhang, Mingchen Shao, Wenjie Tian, Tianlun Zuo, Longhao Li, Lei Xie
Abstract
Automating clinical documentation from long-form doctor-patient conversations remains challenging for modern audio-language models. While cascaded ASR systems perform well, end-to-end (E2E) models often struggle with information loss and hallucinations on extended audio. For the BeTraC 2026 challenge, the ASLP team presents a fully E2E multimodal system that generates structured SOAP notes directly from audio, bypassing intermediate transcripts. We constructed a 1.41-million-sample multi-task corpus and applied a multi-stage pipeline: domain pre-training, supervised fine-tuning, and reward optimization. Evaluating the architecture under both Lightweight (3B) and Heavyweight (30B) constraints reveals that each training stage progressively enhances performance. Furthermore, scaling to 30B parameters substantially boosts concept extraction and summarization quality. Ultimately, our E2E systems consistently outperform representative cascaded ASR+LLM baselines, proving the efficacy of direct multimodal optimization for clinical documentation.