U-shaped split learning improves privacy and accuracy in visual question answering
USPLIT-VQA: U-Shaped Split Learning for Visual Question Answering with Contribution-Aware Weighted Aggregation
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Visual Question Answering (VQA) systems help computers answer questions about images, but they often require gathering all user data in one place, raising privacy concerns. The authors designed a new way called USPLIT-VQA where the client’s device keeps sensitive information and only shares some parts of the processing with a server. They also developed a method to reduce the impact of harmful users trying to mess up the system. Their tests show better accuracy and lower memory and communication needs compared to some existing methods, while better protecting privacy.
What this means in practice
- •For mobile app developers: Build privacy-preserving VQA apps that keep images and questions on user devices while offloading heavy computation to servers.
- •For healthcare it teams: Implement secure VQA tools for medical imaging that protect patient data by processing sensitive input locally and aggregating model updates safely.
Authors
Md Khalid Syfullah, Alvi Ataur Khalil
Abstract
Visual Question Answering (VQA) systems, jointly interpreting images and natural language queries, hold significant promise across many domains, yet the privacy-sensitive nature of user data creates a fundamental barrier. Centralized training requires access to all data, while federated learning requires each client to host the full model. We propose USPLIT-VQA, a U-shaped split learning framework for privacy-preserving VQA in which each client retains the initial layers and the classification head while the server hosts the computationally heavy intermediate layers, keeping raw inputs and labels on the client device. We further introduce Contribution-Aware Weighted Aggregation (CAWA), a gradientsimilarity-based client scoring mechanism designed to reduce the influence of malicious updates. Experiments on four VQA datasets (VQA-RAD, SLAKE, PathVQA, and VizWiz) with two backbones show accuracy gains over Federated Learning for the Custom model and reduced accuracy for BiomedCLIP under the evaluated fixed split, alongside client memory reductions of up to 5.8X and communication reductions of up to 10.8X. With one malicious client, CAWA reduces the attacker's influence by over 98%, while experiments at higher corruption levels identify its limitations. Reconstruction experiments further show lower inversion quality under the evaluated attacks.