Defending Against Harmful Supervision Hidden in Benign Samples
2026-06-29 • Cryptography and Security
Cryptography and SecurityArtificial Intelligence
AI summaryⓘ
The authors point out that current protections usually catch harmful content only when it’s clearly part of the training data. They introduce a new problem called Embedded Attack, where bad information is sneaked into harmless-looking training examples, making it hard to spot. To fix this, they propose a method named Dual-Reference SFT (DR-SFT) that improves how models are fine-tuned by using a special token-level approach, which helps reduce harmful effects that simple data filtering might miss.
fine-tuningharmful contenttraining dataEmbedded AttackDual-Reference SFTDPOcontrastive learningtoken-level regularizationguardrailsQA pairs
Authors
Bang An, Yibo Yang, Dandan Guo, Ebtisam Alshehri, Carlos Hinojosa, Bernard Ghanem
Abstract
Existing defenses are effective when harmful content is explicitly mixed into downstream fine-tuning data, but crafted samples can instead hide harmful supervision inside benign tasks. We propose Embedded Attack, where harmful QA pairs are embedded within benign training samples, and show that representative guardrails often fail to detect them at the example level. To address this, we propose Dual-Reference SFT (DR-SFT), which adapts DPO-style contrastive objective design to SFT through token-level regularization, mitigating harmful fine-tuning beyond coarse data filtering.