Hidden risks emerge when combining LoRA adapters in AI image generation
LoRango: It Takes Two LoRAs to Unlock Hidden Behaviors in Diffusion Models
Cryptography and SecurityComputer Vision and Pattern Recognition
Summary
AI image generation models can be customized using small add-ons called LoRA adapters. This paper shows a security concern: two separate adapters that look safe on their own can secretly work together to change the images generated in unexpected ways. The authors created a method called LoRango that uses pairs of adapters to hide commands that only activate when loaded together, without needing any special changes to the AI model or user prompts. This means just checking one adapter at a time isn't enough to make sure combining them is safe.
What this means in practice
- •For ai system security teams: Evaluate the safety of combined LoRA adapters to prevent hidden malicious behavior in image generation pipelines.
- •For creative ai developers: Improve trustworthiness by auditing adapter combinations to avoid unintended image manipulations when customizing tools.
Authors
Jin Wei, Rundong Li, Ruihao Yang, Yikai Wang, Xiaoyuan Duan, Jianxiong Wu, Yanbo Wang, Chang Xu, Lingyun Zhang, Zhuyang Yu, Ping Chen, Jun Dai, Xiaoyan Sun
Abstract
Users commonly combine multiple Low-Rank Adaptation (LoRA) adapters to personalize images with different subjects, styles, and visual attributes. Yet inspecting adapters individually does not establish the safety of their composition. We identify and characterize a pair-conditioned attack in text-to-image diffusion: individually useful and benign-appearing adapters redirect image generation when co-loaded with a specifically matched partner, whose identity serves as the trigger. We introduce LoRango to realize this attack through complementary Signature and Payload adapters. The Signature writes a pair-specific code into intermediate carrier representations, while the Payload uses code-selective responses and opposing signal/reference branches. These branches approximately cancel for standalone adapters and mismatched pairs; matched code-reader alignment breaks cancellation within native GEGLU blocks and releases the programmed action. Both adapters are exported as ordinary static LoRA files compatible with standard loaders, requiring no prompt trigger or base-pipeline modification. LoRango achieves matched-pair attack success rates of 97.9\% on SD v1.5 and 98.7\% on SDXL, compared with 2.8--4.6\% when implanted adapters are loaded individually. Further experiments evaluate pair selectivity, standalone fidelity, robustness to deployment variations, and applicability across denoiser architectures. These findings show that individual-adapter inspection is insufficient to assess the security of multi-LoRA personalization and motivate auditing adapter compositions.