Semantics aware backdoor removal preserves image details and meaning
SemPurify: Semantics-Aware Data Purification for Black-box Backdoor Defense
Computer Vision and Pattern Recognition
Summary
Backdoor attacks sneak secret triggers into training images to fool AI without changing its usual behavior, making the attacks hard to spot. Current defense methods try to remove these triggers but can blur or distort the images, harming their usefulness. The authors propose SemPurify, which uses understanding of both low-level and high-level image meanings to better restore images while removing backdoor triggers. Their experiments show SemPurify effectively cleans poisoned images without losing important image details or meanings needed for other tasks.
What this means in practice
- •For machine learning engineers: Clean poisoned training data while preserving semantic content to improve model robustness against backdoor triggers.
- •For security operations teams: Detect and neutralize backdoor attacks in deployed AI systems without degrading input image quality.
Authors
Jiawei Yu, Syed Irfan Ali Meerza, Yi Wu, Amir Sadovnik, Jian Liu
Abstract
Backdoor attacks have emerged as a critical security threat in machine learning, where adversaries embed triggers within poisoned training samples to manipulate a model's predictions on targeted inputs while leaving its general behavior unaffected. This stealthy nature makes backdoor attacks difficult to detect, as the model functions normally until activated by a specific malicious trigger. Current purification-based defenses use image transformations (e.g., blurring) to disrupt trigger patterns but often degrade image quality. Even with advanced restoration techniques like diffusion models, these approaches can still alter visual or semantic information, reducing utility for downstream tasks. To address this, we propose the first semantics-aware data purification framework for black-box backdoor defense, SemPurify, which explicitly incorporates multi-level semantics (i.e., low-level semantic embeddings and high-level image tags) within a text-to-image (T2I) framework to guide image restoration during purification. Using this semantic information, the T2I model subsequently refines the transformed samples, helping preserve visual details and semantic information during restoration. Extensive experiments spanning multiple backdoor scenarios, i.e., classification models, CLIP-style encoders, and generative models, demonstrate that SemPurify reliably removes backdoors while effectively preserving semantic information for downstream tasks.