Open source text to image models spread harmful services widely
Monet: Measuring the Ecosystem of Open-Source Text-to-Image Models Tailored for Harmful Services
Cryptography and Security
Summary
Some text-to-image AI models shared publicly online are made for harmful purposes, like spreading unsafe content or evading rules. The authors studied nearly 24,000 such models, called Monets, across major platforms and found they often move between sites to avoid bans. These models can be linked to harmful commercial activities and reach users through popular code repositories and services. This shows that stopping harmful AI models needs cooperation between platforms, not just rules on one site.
What this means in practice
- •For model platform operators: Detect and track harmful AI models spreading across multiple hosting sites to improve coordinated content governance.
- •For security teams: Identify AI models connected to commercial scams and unsafe content to mitigate risks in deployed AI services.
Authors
Zihao Wang, Jiacen Xu, Zilong Lin
Abstract
The open-source text-to-image (T2I) ecosystem enables rapid model development and sharing, but also hosts models intentionally tailored for harmful services, which we call Monets. Prior work has examined specific types of harmful T2I models on individual platforms, but a Monet does not exist in isolation. The broader Monet ecosystem, spanning model characteristics, cross-platform propagation, governance evasion, monetization, and downstream deployment, remains poorly understood. In this study, we present the first systematic, ecosystem-level measurement of Monets. Grounded in the policies of real-world model hubs, we construct a taxonomy of ten harmful service categories and identify 23,947 Monets across eight major T2I model hubs, with the most popular exceeding 19 million downloads. While some developers employ anti-theft mechanisms against unauthorized re-uploading, Monets propagate across platforms at scale, with 40.76% mirrored across hubs. Such propagation further enables governance evasion via cross-platform archiving, keeping 11.99% of Monets accessible after bans on their original platforms, alongside other evasion strategies including keyword obfuscation and model-level safeguard circumvention. Monets also anchor coordinated commercial campaigns---one spanning 668 models with 914 completed commissions and another advertising gray-market account-farming service---and reach users through GitHub projects and inference APIs, raising downstream child safety concerns. These findings expose the limitations of platform-siloed defenses and highlight the need for cross-platform threat intelligence, coordinated governance, and technical safeguards.