UltraText Bench tests how well AI models generate dense bilingual text images
UltraText Bench: A Comprehensive Bilingual Benchmark for Evaluating Visual Text Rendering in Image Generation
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Creating images with lots of text, especially in both English and Chinese, is hard for AI image generators because the text needs to be clear, accurate, and correctly placed. The authors built UltraText Bench, a big test set of images with multiple text areas and exact details to see how well AI models do this job. They also developed a way to automatically score the images on how clear, correct, and well-placed the text is. Their tests show that different AI models have different strengths and weaknesses depending on the difficulty of the text and the language.
What this means in practice
- •For ai development teams: Evaluate and improve AI image generators' capability to create complex images with dense bilingual text accurately and clearly.
- •For software quality assurance teams: Use UltraText Bench to systematically test how well image generation software renders multiple text regions with precise content and placement.
Authors
Deyuan Liu, Yihao Hu, Jingxuan Zhang, Xingying Li, Jun Xie, Jiacheng Liu, Jungang Li, Yu Huang, Xuanyi Liu, Yue Ding, Zecheng Wang, Lei Zhao, Mingda Wang, Zhenglin Cheng, Peng Sun, Tao Lin
Abstract
Dense visual text requires image generators to reproduce long strings across multiple regions with correct placement and legibility. As short-string rendering improves, evaluation must test sustained performance across more demanding scenes. We introduce UltraText Bench, a bilingual benchmark for prompt-only generation of dense visual text. It contains 432 prompts spanning 24 real-world scene categories and three difficulty levels, split equally between English and Chinese. Each human-reviewed prompt supplies exact strings for four to twelve text regions, paired with structured references for their content, placement, and visual attributes. We use the Q-Judger vision-language model to assess each image against the complete reference, reporting text fidelity, text clarity, spatial quality, and scene quality. Across 24 model configurations, these dimensions reveal different strengths: Z-Image-Turbo gains 3.81 clarity points over Z-Image-Base while losing 14.76 fidelity points under the reported settings. Performance also varies with workload; Qwen-Image-2512's English composite falls from 86.50 at L1 to 42.86 at L3. Ten participants took part in human evaluation of the automatic scores. Repository: https://github.com/LINs-lab/UltraText_Bench.