TraceGuard detects poisoned data in image text training datasets
TraceGuard: Adaptive Multimodal Poison Filtering through Cross-Feature Rank Agreement
Cryptography and SecurityMachine Learning
Summary
Training AI models with images and text from the internet can be risky because attackers can hide harmful examples that trick the model. The authors study what makes a small set of bad data powerful enough to affect training. They develop TraceGuard, a tool that spots suspicious examples by looking for agreement among different patterns in the data without needing to train the model first. Their tests show TraceGuard removes most bad examples while keeping most good ones, making AI models safer to train.
What this means in practice
- •For machine learning engineers: Remove poisoned examples automatically from image-text datasets to improve the safety of training vision-language models.
- •For security teams in ai companies: Detect stealthy data poisoning attacks in AI training corpora without needing prior knowledge of the attack or poison rate.
Authors
Haoyang Li, Yaxin Xiao, Linyan Dai, Jiawen Fu, Zi Liang, Jason Xue, Qingqing Ye, Haibo Hu
Abstract
Multimodal training relies on image-text corpora collected from external sources, creating opportunities for attackers to poison the data. Stealthy attacks can preserve plausible image-text pairs while concealing the differences used by detectors, so apparently clean data can still redirect the trained model. We therefore ask which properties a poison set must preserve for the attack to remain effective. A small poison set must still exert enough collective influence during training to induce the attacker's target behavior. We analyze this influence in terms of how often an attack pattern occurs and how strongly the examples carrying it jointly affect the model. This analysis motivates six corpus-level features that examine cross-modal neighborhoods, recurring text, and changes after text-span erasure without training the victim model. We introduce TraceGuard, an adaptive rank-based filtering method that uses agreement among complementary feature rankings to identify suspicious examples. It refines the selected set through shared patterns and adapts the removal threshold to each corpus without knowing the attack or poison rate. Across 19 attack configurations spanning image-text learning, generative vision-language model fine-tuning, and encoder-transfer tests, TraceGuard removes an average of 98.4% of poisoned examples and 5.4% of clean examples. After training on the filtered corpora, the residual attack metric is at most 1% in 13 configurations. Matched-removal controls and ablations support the contributions of sample selection and adaptive removal. Stress tests also identify detection failures under adaptive attacks and unnecessary removal on poison-free corpora.