Video text editing improved by aligning glyphs with text movement
Enhanced Video Text Editing with Trajectory-Aligned Glyph Rendering
Computer Vision and Pattern Recognition
Summary
Editing text in videos is hard because the new text needs to match the movement and angle of the original in every frame, which existing video diffusion methods struggle with. The authors created a way to guide the editing using exact shapes of letters matched to the text’s path and a new method to check mistakes in letter strokes more carefully. They also made a new set of 288 video clips with annotated text to test such methods. Their approach makes text changes more accurate and keeps the background better than previous methods and commercial tools.
What this means in practice
- •For video production teams: Enable precise and stable replacement or addition of text in video scenes without affecting background quality.
- •For graphic design software developers: Incorporate trajectory-aligned glyph rendering to improve text editing tools for dynamic video content.
- •For advertising agencies: Create accurate video ads with custom text edits that follow the scene perspective and motion.$Commercial implications: Allows development of video editing products that deliver precise text overlays matching scene dynamics, appealing to marketing clients.
Authors
Shulian Zhang, Xiangyu Shu, Wenbo Li, Jian Chen, Yong Guo
Abstract
Video text editing aims to replace or add text in a video while keeping the rest of the video unchanged, which requires the edited text to be correct in every frame and to move coherently with the scene. Despite the remarkable progress of video diffusion models, they struggle to reproduce exact stroke structures and often produce garbled or wrong characters, especially for characters with complex strokes. To address this, we propose a trajectory-aligned glyph rendering reference that provides explicit per-frame glyph guidance following the position and perspective of the text, and a depth-normalized recognizer feature supervision that supervises the generated text on multi-depth features of a frozen text recognizer with per-depth normalized errors, targeting stroke errors overlooked by the diffusion loss. We further build VTEdit, a benchmark of 288 real-scene clips with 440 annotated text trajectories covering text replacement and text addition, which will be publicly released to facilitate future research. Experiments on VTEdit show that our method outperforms image text editing methods, video editing methods, and commercial models in text accuracy and background preservation, achieving a sentence accuracy of 0.9408, and receives the highest preference in a user study.