Smartphone skin lesion classification improves with grouped clinical data fusion
GAMF-Net: Grouped Availability-Aware Multimodal Fusion for Smartphone Skin Lesion Classification
Computer Vision and Pattern Recognition
Summary
Classifying skin lesions using just smartphone images can miss important health information. The authors propose GAMF-Net, a new method that groups patients' clinical data into categories and combines them with images only when available. This approach helps the system better decide if a lesion is dangerous by balancing evidence from both images and clinical details. Their tests show it performs better than methods using images alone or mixing data without grouping.
What this means in practice
- •For mobile health app developers: Develop apps that classify skin lesions more accurately by merging patient clinical data groups with smartphone images only when available.$Commercial implications: Enables more reliable skin lesion diagnostic apps selling enhanced accuracy to healthcare providers and consumers.
- •For hospital data teams: Integrate grouped clinical data with image analysis to improve automated skin lesion triage systems without losing performance when data is missing.
Authors
Yue Zhuo, Jingjia Yan, Fangyu Liu, Lingjun Liu, Zongjun Chen, Zhisheng Luo, Zhonghua Xie
Abstract
Smartphone clinical images provide a low-cost data source for computer-aided skin lesion classification in resource-constrained settings, but images alone cannot fully capture patients' medical histories and lesion symptoms. To address the lack of structured organization among heterogeneous clinical variables, the ambiguity between observed and imputed values, and the risk that clinical information may interfere with visual evidence, we propose GAMF-Net, a grouped availability-aware multimodal fusion network. Ten clinical variables are divided into three semantic groups---demographics, history-related risk, and lesion presentation---and each group jointly encodes field values and field-level availability masks. The image branch produces the base classification logits, while each clinical group generates image-conditioned class-level residual evidence. Only available groups participate in availability-normalized equal-weight fusion; when all clinical information is unavailable, the model reduces exactly to the image-only pathway. We evaluate the method on PAD-UFES-20 under a patient-disjoint five-fold protocol with separate training, validation, and test partitions. GAMF-Net achieves an Accuracy of $74.17\pm2.85\%$, a balanced accuracy (BACC) of $70.11\pm2.11\%$, a Macro-F1 of $68.85\pm0.85\%$, and a Macro-AUROC of $93.42\pm1.20\%$, outperforming all image-only and multimodal baselines reimplemented under a unified protocol. Compared with the strongest baseline, MetaBlock, the corresponding improvements are 2.37, 2.59, 4.14, and 1.02 percentage points, respectively. Ablation results show that clinical information substantially improves class-balanced performance, constrained equal-weight fusion is more effective than dynamic fusion, and the lesion-presentation group provides the most informative clinical complement.