RADIANCE: Relative Adaptive Denoising with IP-Adapter for Novel Concept Enhancement
2026-07-06 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors address a common problem in text-to-image models where rare or unusual combinations of objects and attributes are not well generated, often missing key elements or mixing them up. They propose RADIANCE, a new method that improves this by monitoring the image creation process and adjusting it on the fly without needing extra training. Their approach uses three main tools to detect and correct imbalances during image generation and special techniques to handle prompts with multiple objects. Tests show that RADIANCE makes images more accurate and visually better without slowing things down much.
text-to-image diffusion modelscompositional balanceCLIPlatent spaceinference feedbackadapter scalingdenoising trajectorycompositional alignmentmulti-object prompts
Authors
Zi-Xiang Ni, Bo-Lun Huang, Teng-Fang Hsiao, Bo-Kai Ruan, Hong-Han Shuai
Abstract
Text-to-image (T2I) diffusion models have achieved striking progress but still struggle to synthesize rare concepts involving unusual attribute-object pairings, often resulting in concept omission or semantic drift where a dominant entity overwhelms the generation. Tracing these failures to a lack of compositional balance during the denoising trajectory, we propose RADIANCE, a training-free framework that treats inference as a closed-loop feedback process. RADIANCE augments pretrained backbones with three modular components: (1) a Compositional Similarity Monitor (CSM) that tracks the emergence of objects and attributes in intermediate latents via CLIP-based feedback; (2) a Bidirectional Scale Controller (BSC) that applies a reactive "restoring force" using positive and negative IP-Adapter scales to rebalance biased trajectories; and (3) a Feedback Guidance Scheduler (FGS) that coordinates these updates across timesteps without additional training. We further extend the framework to multi-object prompts via Delayed Adapter Activation (DAA) and Layer-wise Alternating Guidance (LAG) to prevent premature concept fusion. By overlapping monitoring and denoising through pipelined execution, RADIANCE maintains competitive latency while significantly enhancing the per-sample success rate and effective throughput. Experiments on RareBench and T2I-CompBench demonstrate that RADIANCE consistently enhances compositional alignment and perceptual quality over state-of-the-art baselines.