Transformer models improve clinical specialty classification from medical notes
Clinical Domain Classification from Medical Transcriptions
Computation and LanguageMachine Learning
Summary
Sorting medical notes into clinical specialties helps organize lots of health information. However, some specialties have far fewer notes, making the sorting less accurate. The authors compared classic machine learning methods and newer transformer AI models like BERT to see which sorted medical transcriptions best. They also tried ways to balance data by making extra examples for the rarer specialties. Their findings show that using transformer models with balanced data works best and trains efficiently, especially with BERT.
What this means in practice
- •For healthcare it teams: Use transformer-based classification to better organize large sets of medical notes by clinical specialty with improved accuracy and efficiency.
- •For medical transcription providers: Enhance transcription services with automated tagging of clinical specialty to speed up review and routing processes.
Authors
Sravani Pottipati, Lakshmikar R. Polamreddy
Abstract
Clinical domain classification plays an important role in organizing and analyzing large volumes of unstructured medical text. However, medical transcription datasets are often highly imbalanced, which can substantially degrade classification performance, particularly for underrepresented clinical specialties. In this work, we present a comparative study of machine learning and transformer-based approaches for clinical domain classification from medical transcriptions. We evaluate six traditional machine learning classifiers---Naive Bayes, Support Vector Machine (SVM), Decision Tree, Random Forest, K-Nearest Neighbors (KNN), and XGBoost---along with two pretrained transformer models, BERT and XLNet, and a few-shot large language model prompting approach. Experiments are conducted on medical transcription data collected from MTSamples, comprising 5,013 samples across 40 clinical specialties. To address severe class imbalance, we investigate two balancing strategies: text augmentation using NLP-based synonym replacement and Synthetic Minority Over-sampling Technique (SMOTE). Experimental results demonstrate that data balancing substantially improves classification performance across the evaluated models. In particular, BERT achieves the highest F1-score of 0.996 on the SMOTE-balanced dataset while requiring lower training time than XLNet. The results highlight the effectiveness of transformer-based representations combined with appropriate data balancing strategies for clinical domain classification and provide a systematic comparison of classical machine learning, transformer models, and few-shot prompting for medical transcription analysis.