Streaming videos add new subjects smoothly without retraining
ReferStream: Interactive Subject Injection for Streaming Video Generation
Computer Vision and Pattern Recognition
Summary
Adding new characters or objects into an ongoing video without messing up what’s already there is hard. The authors created ReferStream, a way to do this by carefully blending new visual information into the current scene as it’s generated, without needing to retrain the video system. They also built a test set called ReferStream Bench that checks how well new subjects fit and old ones stay consistent. Their method performs better than others, including those that know all new subjects beforehand.
What this means in practice
- •For video streaming developers: Enable live integration of new visual subjects into ongoing video streams without interrupting or retraining the generator.
- •For augmented reality content creators: Add or update visual elements dynamically in AR videos while maintaining scene consistency and history.
Authors
Shuolin Xu, Guanyuan Li, Boyu Li, Kelu Yao, Xiaogang Xu, Hongchuan Yu, Shuicheng Yan, Xiaobin Hu
Abstract
Injecting new subjects naturally during streaming video generation requires integrating newly supplied visual references into the existing video while preserving established subjects and the scene. Encoding references independently of the current scene provides little guidance for this integration, while keeping their features in persistent memory can interfere with established context. We propose ReferStream, a framework that requires no additional training and combines reference construction using the current scene with transient cache injection. Joint Latent places reference latents in target regions of the latest scene latent, allowing reference features to be extracted in context. Cache Restoration uses these features to condition the injection chunk, then restores the overwritten global cache entries while retaining the newly generated history. This procedure incorporates new visual conditions without retraining the generator or revising emitted frames. We further introduce ReferStream Bench, a new benchmark with scheduled reference arrivals, evolving instructions, and explicit annotations of subject presence for InShot and MultiShot generation. Evaluation follows the event schedule and measures both the fidelity of new subjects and the preservation of established subjects, alongside video quality, text adherence, and count accuracy. Experiments on ReferStream Bench show higher scores for text adherence, subject similarity, and count accuracy than the compared baselines, including offline methods given all references in advance. Project page: https://xushuolin.github.io/ReferStream/.