Multimodal models show mixed accuracy classifying breast ultrasound images
Can Multimodal Decision Models Classify Breast Ultrasound Images? Benchmarking CLEF and CLEF-Flash Across Prompts and Datasets
Computer Vision and Pattern Recognition
Summary
Classifying breast ultrasound images correctly is important for detecting cancer. The authors tested two computer models, CLEF and CLEF-Flash, on three different sets of breast ultrasound images without changing the models. They found that model performance depends a lot on which dataset and instructions (prompts) are used. Sometimes the models made more mistakes labeling benign (non-cancerous) images as malignant (cancerous). This means these models should be carefully evaluated on each specific dataset before being used in medical diagnosis.
What this means in practice
- •For clinical imaging specialists: Use benchmarked multimodal models as a tool to classify breast ultrasound images with known performance depending on dataset and prompt choice.
- •For medical ai developers: Design breast ultrasound classification systems by selecting appropriate prompts and models based on dataset-specific performance evaluations.
Authors
Tran Cao Minh, Hai Tran, Ha Minh Tan
Abstract
Multimodal decision models return probabilities over predefined alternatives, offering a structured interface for image classification. This study benchmarks inference with CLEF and CLEF-Flash on 2818 breast ultrasound images from Dataset B, BUSI, and BUS-BRA. Dataset B and BUS-BRA use binary benign-malignant classification, whereas BUSI includes normal, benign, and malignant categories. Model parameters remain unchanged, and original images are submitted without user-defined preprocessing. Three prompts provide minimal labels, an explicit classification task, or sonographic criteria. We report accuracy, macro F1, balanced accuracy, macro precision, throughput, and latency. CLEF-Flash with the clinical prompt achieves the highest accuracy on all three datasets: 74.85% on Dataset B, 66.28% on BUSI, and 51.63% on BUS-BRA. On BUSI, however, CLEF with the task-guided prompt achieves the highest macro F1 (64.51%) and balanced accuracy (68.72%). CLEF-Flash has lower latency in every matched comparison. Clinical instructions improve CLEF-Flash accuracy relative to the minimal prompt but reduce CLEF accuracy on all datasets. Confusion matrices reveal frequent benign-to-malignant errors, particularly on BUS-BRA, where all configurations fall below the 67.63% accuracy of a majority-class predictor. These findings demonstrate dataset-and prompt-dependent performance and motivate class-specific evaluation before applying multimodal decision models to breast ultrasound analysis.