Defending AI agents from hidden harmful behaviors after fine-tuning
Defending Against Dormant Poisoning Attacks Across Language and Multimodal Agents
Cryptography and SecurityComputation and LanguageComputer Vision and Pattern Recognition
Summary
Some AI models can secretly hide harmful behaviors that only appear after they are slightly changed or updated. The authors show these hidden problems can cause AI agents to act in harmful ways, even after updates meant to be safe. They developed a new defense that uses the model’s own safe responses at release as a guide to detect and stop harmful changes during updates. Their method reduces harmful actions dramatically while keeping the AI useful and less likely to refuse harmless requests. This works across language and multimodal AI agents, improving trust in systems that learn from later fine-tuning.
What this means in practice
- •For ai safety engineers: Reduce hidden harmful behaviors in language and multimodal AI agents after fine-tuning updates.
- •For software developers of ai assistants: Maintain utility while preventing AI assistants from generating harmful or refused responses during continual updates.
- •For security teams in ai service providers: Detect and mitigate contamination of deployed AI systems caused by dormant poisoning attacks at fine-tuning time.$Commercial implications: Enables security solutions that safeguard fine-tuned AI agents against stealthy poisoning attacks, protecting user trust and compliance.
Authors
Wonjun Lee, Vaidehi Patil, Zaid Khan, Yue Zhang, Hyunji Lee, Suhyun Kim, Mohit Bansal
Abstract
Recent work shows that attackers can implant dormant malicious behaviors that remain hidden at release but emerge after benign fine-tuning. As foundation models are used as agents, whether dormant poisoning extends to harmful agent actions and multimodal settings remains unclear. We show two agentic risks: (1) harmful behaviors activated by benign fine-tuning can extend to unseen actions; (2) multiple harmful behaviors can coexist and be jointly activated by benign downstream fine-tuning. Existing defenses often rely on external safe responses, which are hard to obtain for agent actions, or a trusted clean reference model. We exploit a key property of dormant poisoning: the attacker must preserve safe behavior at release to conceal poisoning, and this behavior can serve as a safety reference. Leveraging this property, we automatically construct a Safety Buffer pairing harmful inputs with the released model's responses and benign versions generated by the same model to remove harmful intent. The Safety Buffer can replace external safe responses in existing defenses, but these defenses do not distinguish updates that weaken safety from those that preserve it, often increasing over-refusal or reducing utility. To address this limitation, we introduce LookAhead Defense, which previews benign fine-tuning updates using the Safety Buffer and penalizes only updates predicted to weaken safe behavior on harmful inputs or increase the likelihood of refusals on benign versions in the Safety Buffer. Across language models, language agents, and multimodal agents, LookAhead Defense reduces harmful behavior in agents from up to 93.9% to at most 14.2% and harmful responses in language models from up to 92.5% to at most 2.0% versus vanilla fine-tuning, while maintaining competitive utility and low over-refusal. It remains effective against other poisoning methods.