InGuard improves safety inside text to image generation models
InGuard: Towards Generalized Inner Guardrail for Safe Text-to-Image Generation
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Text-to-image AI can create unsafe or inappropriate pictures based on user prompts. The authors address this by designing a safety system that works inside the image generation process itself. Their method detects risky prompts early and adjusts the AI’s understanding to create safe images instead of refusing prompts. They also stop unsafe generations early, saving time and resources. Tests on multiple models showed this approach matched or beat traditional safety checks while causing fewer problems with normal prompts.
What this means in practice
- •For text-to-image developers: Integrate safety checks inside the model pipeline to reduce unsafe content generation and improve efficiency.
- •For content moderation teams: Use early detection in generation to halt unsafe image creation and reduce content review costs.
- •For digital marketing platforms: Offer safe text-to-image generation services that minimize refusal rates and maintain creative flexibility.$Commercial implications: Enables building safer image generation products appealing to broad customers with reduced content risk.
Authors
Zeyu Wang, Xiaodan Li, Zhiwen Li, Yuefeng Chen, Hui Xue
Abstract
Modern text-to-image (T2I) models generate high-quality images from arbitrary user prompts, yet they can just as easily produce not-safe-for-work (NSFW) content. Conventional outer guardrails consist of two components: a prompt classifier that checks for risk before generation, and a post-hoc image classifier that checks the fully generated image. In this design, both classifiers operate outside the generation pipeline and do not use the model's own representations. This separation can limit prompt-screening accuracy, while the image-side check runs only after the full generation cost has been spent. Moreover, a flagged prompt can only be rejected, even when it could be adjusted to produce a safe image. In this work, we propose the Inner Guardrail (InGuard), a safety framework that works inside the pipeline on the model's own representations, leaving base-model parameters untouched. First, a risk classifier grades each prompt as unsafe, risky, or benign based on the text encoder's embeddings, with no external language model. Second, SAGE (Soft-gated Asymmetric Guardrail for Embeddings) modifies the embeddings of risky prompts, aiming to return a safe image instead of a refusal. Third, a latent detector checks the one-step clean latent estimate midway through denoising, reaching nearly image-level performance and halting generation when risk is detected. We also construct the RevGen Safety Benchmark to evaluate T2I safety under realistic conditions: 10,000 prompts built through real-image reverse generation, with a rewriting step that supplies controlled intellectual-property (IP) characters, covering graded porn/gore risks, categorical IP risks, and benign negatives. Across five open-weight T2I models, InGuard reaches 97.9-98.8% safety rate, matching or exceeding the outer guardrail, with 57.5-73.5% less benign disturbance, ~3.7x fewer parameters, and 50-55.6% of denoising steps skipped.