ECGQuest: Benchmarking and Fine-Tuning Language Models for Electrocardiography
2026-08-31 • Computation and Language
Computation and LanguageInformation Retrieval
AI summaryⓘ
The authors created ECGQuest, a large set of true/false questions based on ECG research papers, to test language models on their understanding of ECG interpretation beyond just looking at ECG images. They tested many popular language models and found that general-purpose models often did better than specialized medical ones, but all models improved when fine-tuned on the ECGQuest data. Fine-tuning smaller models helped them perform almost as well as much bigger commercial models. The dataset and results offer a new way to measure how well AI models can understand the complex knowledge needed for ECG interpretation.
Electrocardiogram (ECG)Language ModelsFine-tuningTrue/False QuestionsGPT-4oZero-shot LearningMedMCQAMedQALow-Rank AdaptationBenchmark Dataset
Authors
Mohammadsina Hassannia, Matthew A. Reyna, Reza Sameni
Abstract
Electrocardiogram (ECG) interpretation requires knowledge of cardiology, electrophysiology, clinical diagnosis, ECG waveforms, signal acquisition, and instrumentation. Existing language-model benchmarks, however, primarily assess broad medical knowledge or interpretation of individual ECG signals and images rather than the broader contextual knowledge required for ECG interpretation. We developed ECGQuest, a literature-grounded resource for evaluating and fine-tuning ECG-specific language models. A GPT-4o-based pipeline generated questions from 23 ECG references and Computing in Cardiology proceedings from 2003-2025. The final dataset contains 10,904 unique True/False questions paired with their negated forms (21,808 Q&A pairs). We evaluated three commercial and 20 open-source language models on a held-out test set in a zero-shot setting. Five open-source models with 7-14B parameters were fine-tuned using Low-Rank Adaptation, with BERT and BiomedBERT included as supervised encoder baselines. Generalization was assessed on ECG-related subsets of MedMCQA and MedQA converted to binary True/False questions using official answer keys. Zero-shot accuracy on ECGQuest ranged from 49.5% to 74.4%, with GPT-5 performing best. General-purpose models outperformed medically specialized models, several models showed strong True/False bias, and encoder baselines performed near chance. Fine-tuning improved all open-source models by 6.5-14.1%. Fine-tuned DeepSeek-R1-Distill-Qwen-14B reached 76.3% accuracy, while a five-model voting ensemble reached 78.5%. On MedMCQA and MedQA, fine-tuning mainly benefited weaker or class-biased models and did not consistently improve strong base models. ECGQuest provides a reproducible benchmark for contextual ECG knowledge and shows that parameter-efficient fine-tuning can make smaller language models competitive with substantially larger commercial models.