Model removes burned in video text while preserving background details

Hint, Not Hole: Localization-Robust Video Overlay Removal

Computer Vision and Pattern Recognition

Summary

Sometimes videos have text like subtitles or logos stuck onto their images, which can be tricky to remove perfectly. The authors created a method where instead of treating the detected text area as a hole to fill, their model sees the detection as just a hint and looks at the whole video to decide what to remove or keep. This way, their model removes text that the detector might have missed, avoids removing parts of the background it shouldn’t, and keeps any actual scene text intact. They also built a streaming system to handle long videos with complicated editing features smoothly.

What this means in practice

Authors

Dongyang Zhang, Wei Zhang, Renjie Zou, Zhiwei Huang

Abstract

Video overlay removal recovers the scene beneath burned-in subtitles, karaoke lyrics, titles, credits, logos and watermarks. Generative video models now fill a known region convincingly, but in practice the region comes from a detector or a segmenter, and it is wrong in both directions: it misses strokes, halos, whole frames and whole objects, and it covers background between characters, around boxes and on genuine scene text. Erasers that treat the mask as a hole, regenerating inside and copying outside, inherit every one of these errors. We present HintErase, which treats localization as a hint, not a hole. The model sees the full, unmasked video together with a hint channel; during training the hint is corrupted the way real localizers fail, while supervision follows the true overlay. The result is a single model that removes overlay the hint missed, keeps background the hint wrongly covered, and is trained to keep genuine scene text even when the hint lands on it. For full-length releases, a streaming pipeline handles interlaced sources, shot cuts, missed and jittering hints, paste-back and audio, with memory independent of the length of the video. This report describes the approach, the pipeline that turns it into full-length deliveries, and qualitative results; video results, with input, hint and output side by side, are on the project page.