Securing Self-supervised Data Curation for Foundation Models Robustness
2026-06-08 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors explain that using self-supervised learning (SSL) to gather large datasets helps train better machine learning models without much human work. However, these automatically gathered datasets can contain harmful fake data, so the authors created a Poisoned Data Detector (PDD) to find bad data before training starts. They combined a pretrained model called ImageBind with traditional classifiers like Random Forest and SVM to detect attacks in various datasets. Their tests showed that the SVM-PDD works best in different scenarios, and their system can quickly adapt to new types of bad data by combining multiple detectors.
self-supervised learningdata curationdata poisoningPoisoned Data DetectorImageBindRandom Forestk-Nearest NeighborsSupport Vector Machinesadversarial attacksfoundation models
Authors
Sandeep Gupta, Roberto Passerone
Abstract
Self-supervised data curation provides a pathway to scaling and improving the generalization capabilities of machine learning models. By leveraging self-supervised learning (SSL) for data curation, the demand for massive training datasets required by foundation models can be effectively met. SSL greatly alleviates the costs associated with annotation and manual dataset curation while minimizing the need for human oversight. However, the integrity of SSL-curated datasets must be rigorously checked, as reliance on anonymous and unvetted external sources can substantially increase the risk of data poisoning. In this paper, we propose a Poisoned Data Detector (PDD), an active defense mechanism designed to ensure the integrity of SSL-curated datasets prior to foundation model training. PDDs are designed using a combination of the pretrained ImageBind model and traditional classifiers, including Random Forest (RF), k-Nearest Neighbors (KNN), Naive Bayes (NB), and Support Vector Machines (SVM). We rigorously evaluated PDDs using 176,200 images from three diverse datasets and three different adversarial attacks encompassing both in-distribution and out-of-distribution scenarios. Notably, SVM-PDD achieves superior performance for both in-distribution (Set3-Set5) and out-of-distribution (TrueFace and 140K RealFace) datasets. Our design demonstrates strong scalability and enables the rapid integration of new adversarial attack detectors through an ensemble approach.