Backdoors in large language model agents can survive fine-tuning and reinforcement learning
Understanding and Enhancing Backdoor Persistency in LLM Agent Post-Training
Cryptography and Security
Summary
Sometimes models people get from outside come with secret hidden commands called backdoors that make them act badly in certain situations. The authors show these hidden tricks often still work even after the models are improved with normal training and special practice sessions. They found ways these hidden tricks stay alive and created a method to make them last even longer. This means people who build AI tools need better checks to find and remove these sneaky backdoors. The authors shared their method and results openly for others to study.
What this means in practice
- •For ai security teams: Detect and mitigate hidden backdoors that can survive typical fine-tuning and reinforcement learning updates in third-party AI models.
- •For software developers using llm agents: Assess risks of inherited hidden behaviors when adapting third-party language models for software development tasks.
Authors
Qiusi Zhan, Nian Lyu, Stephanie Ding, Arnav Mehta, Xander Davies, Daniel Kang
Abstract
Developers can build LLM agents by adapting third-party models through benign post-training. We study a supply-chain threat in which an attacker supplies a model with a backdoor: hidden behavior that produces malicious outputs when a particular input pattern appears. Focusing on software-engineering agents, we ask whether such backdoors survive the developer's supervised fine-tuning (SFT) and subsequent task-level reinforcement learning (RL). We observe that benign SFT substantially reduces attack success, but subsequent RL often preserves the residual behavior and sometimes even increases attack success. Our analysis of backdoor erosion during SFT identifies two factors that may favor survival: initial backdoor strength and gradient compatibility with benign training. These factors motivate PersistBD, which refines an already-backdoored model before release to improve its persistency through the benign post-training process. On Qwen2.5-Coder-7B, PersistBD raises attack success from 20% to 74% after SFT and from 20% to 76% after SFT-RL, while maintaining comparable benign task performance. Together, our results show that backdoors can remain active through benign post-training and that adversaries can deliberately increase their persistence. This highlights a supply-chain risk for AI developers and motivates stronger techniques for detecting and mitigating inherited backdoors when adapting third-party models into agents. Our code is available at https://github.com/uiuc-kang-lab/PersistBD.