Video generation models struggle to correctly render scene text
VTR-Bench: A Systematic Benchmark for Evaluating Visual Text Rendering in Video Generation
Computer Vision and Pattern Recognition
Summary
Videos created by artificial intelligence can look very real, but they often mess up text that appears in the scene, like signs or labels. The authors made a special test called VTR-Bench to check how well these models can show text accurately in different video scenes, like ads or science videos. They also built a system that helps improve videos by guiding the generation process with feedback. Their tests showed that even top models make many text mistakes, highlighting this as a big challenge in video AI.
What this means in practice
- •For video production teams: Use VTR-Bench to systematically assess and improve the accuracy of text displayed in AI-generated videos for marketing or informational content.
- •For media localization teams: Evaluate and refine automated video translation and subtitle insertion quality by measuring and improving text rendering fidelity in generated videos.
Authors
Yu Huang, Jungang Li, Zhiyuan Wang, Yonghua Hei, Song Dai, Jiayu Yang, Deyuan Liu, Xiang Zheng, Xiaoshuang Shi, Hao Cheng, Kaidi Xu
Abstract
Recent video generation models can produce highly realistic videos from natural language instructions, with visual quality approaching cinematic standards. Existing evaluation benchmarks, however, predominantly assess visual quality, aesthetic appeal and physical plausibility, while paying limited attention to text, an essential medium for conveying information in everyday scenes. A generated video may appear visually compelling and feature lifelike subjects, yet still render the text within the scene incorrectly. To address this overlooked dimension, we introduce \textbf{VTR-Bench}, a systematic benchmark for evaluating the \textbf{V}isual \textbf{T}ext \textbf{R}endering capabilities of video generation models. VTR-Bench situates text within concrete application scenarios, such as advertisements and scientific videos, with 300 carefully constructed prompts spanning five scenario categories. We develop an automated evaluation pipeline with human alignments that separately assesses text fidelity through carrier-specific transcription and scene and motion requirements through a prompt-specific chain of query. Beyond evaluation, we introduce a \textbf{Keyframe-Guided Agentic Framework} in which a Director agent coordinates image and video generation with visual evaluation, guiding iterative refinement and candidate selection through visual feedback. Experiments on 11 state-of-the-art models reveal widespread difficulties in accurately rendering scene text, with the best-performing model recording an overall word error rate (WER) of 0.250. We further analyze text rendering failures to characterize the challenges faced by current video generation models. These findings highlight visual text rendering as a key challenge for video generation and demonstrate a practical path toward improvement. Code is available at https://github.com/hardenyu21/VTR-Bench.