Federated learning method improves few-shot image classification with smaller models
FedSSMCoOp: SSM Encoders for light-weight Federated Prompt Learning for Few-shot Classification
Computer Vision and Pattern Recognition
Summary
Many vision-language models use simple similarity metrics that miss important details between images and text, especially in sensitive fields like medical imaging where data is private and rare. The authors propose a new approach called FedSSMCoOp that allows different hospitals to train a shared model without sharing data, using specialized components to better connect image and text information. This method is lighter and performs steadily across several medical image datasets. It also removes the need for larger external language models to align features, saving computing resources.
What this means in practice
- •For hospital data teams: Implement privacy-preserving federated learning to improve medical image classification when labeled data is scarce and varies by site.
- •For biomedical ai developers: Build lightweight, few-shot classification models integrating text and image data without relying on large external language models.
Authors
Ankita Das, Ambarish Parthasarathy, Sumohana S. Channappayya, C. Krishna Mohan
Abstract
Vision-Language Models (VLMs) have shown strong performance across a wide range of downstream vision tasks, thanks to the complementary information contained in the respective domains. Despite the performance gains, most of these approaches rely on aligning these domains using the cosine similarity metric, which fails to capture token-level structure and cross-modal interactions prior to the classification stage. This is especially critical in biomedical applications under federated constraints, where data sharing is restricted, labeled data is scarce at each site, and it differs widely across institutions, leading to substantial statistical heterogeneity. To overcome this issue, we propose FedSSMCoOp, a federated few-shot image classification framework that enables multimodal learning while preserving data privacy. With the help of the SSM-based Vision Mamba and Cross Mamba blocks, and by optimizing only the soft-prompt and communication-prompt updates in the federated setting, the framework prioritizes both computation and performance. Importantly, this eliminates the need to use an external Large Language Model (LLM) for feature alignment. The framework is further trained and evaluated on various biomedical image datasets, and its performance is assessed. The proposed framework delivers stable performance relative to the baselines and is, on average, 1.96 times lighter. The corresponding script will be made available soon.