PRISM: Prompt Refinement via Image-grounded Self-rewarding Mechanism for Text-to-Image Generation
2026-07-27 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors created PRISM, a system that helps improve text-to-image models by checking how well the pictures match the words given. Instead of just changing the text prompts, PRISM looks at the images themselves to see if they are clear, pretty, and close to what humans would want. It uses a special way to give feedback based on multiple qualities and then adjusts the prompts to make better images over time. Their tests show this method helps the model produce higher quality and more accurate images.
text-to-image generationprompt optimizationvisual language model (VLM)semantic consistencyaesthetic qualityself-rewarding optimizationChebyshev rewardmulti-task learningimage-grounded feedbackprompt refinement
Authors
Guo Tang, HongJie Luo, Tianxu Wang, Ying Zhang, Hao Wang
Abstract
Text-to-image generation models can synthesize high-quality images from natural language descriptions, but their performance remains highly sensitive to prompt formulation. Existing prompt optimization methods mainly rely on text-side rewriting, prompt expansion, or external reward signals, offering limited image-grounded diagnosis and weak support for learning reusable optimisation policies. In this paper, we propose PRISM, a Prompt Refinement framework via Image-grounded Self-rewarding Mechanism. PRISM closes the prompt-image-feedback loop by interpreting generated images with structured visual diagnosis and scoring them along semantic consistency, aesthetic quality, and human preference alignment. It first initializes a unified VLM through multi-task supervised fine-tuning, and then improves the prompt policy via self-rewarding optimization with a hybrid ideal-point and Chebyshev reward. Extensive experiments show that PRISM improves holistic image quality and fine-grained semantic alignment, while providing interpretable feedback for targeted prompt refinement. The code is available at https://anonymous.4open.science/r/PRISM-FF81.