Vidu S2 enables real-time editable interactive video with avatars
Vidu S2: Real-Time Interactive, Editable, and Spatial Video Generation
Computer Vision and Pattern RecognitionMachine Learning
Summary
Creating and changing videos in real time is hard. The authors show Vidu S2, a tool that can generate live video of digital characters at 720p quality and lets users change styles, clothes, characters, or backgrounds while the video plays. It improves on an earlier version by allowing fast updates and more complex user commands like dancing. This system lets users interact instantly with video content, opening up new possibilities for live editing and avatar use.
What this means in practice
- •For video game developers: Generate and manipulate real-time digital character animations within games for dynamic avatar interactions and customizable appearances.
- •For live streaming teams: Provide interactive live video streams with customizable avatars and on-the-fly style or background changes to enhance viewer engagement.$Commercial implications: Enables sale of advanced live streaming software that offers real-time avatar and scene customization for content creators.
Authors
Jintao Zhang, Kai Jiang, Jintao Chen, Xu Wang, Deyuan Liu, Jungang Li, Dechuang Chen, Ming Lin, Jingjiang Zhou, Haopeng Jin, Qi Jia, Xiaohang Wang, Yaole Wang, Zhanqiang Zhang, Ran Li, Zhengkun Huang, Shuyue Xiong, Yuji Wang, Zikun Dai, Hui He, Yang Luo, Mang Ning, Weiqi Feng, Chengyang Ye, Xinyue Lin, Min Zhao, Hongzhou Zhu, Hengkai Tan, Zeyuan Wang, Chendong Xiang, Kaiwen Zheng, Zhijie Deng, Fan Bao, Jianfei Chen, Jun Zhu
Abstract
We present Vidu S2, which comprises Vidu S2-Avatar, a real-time interactive digital-character model, and Vidu S2-Editing, a real-time video editing model. Moreover, we explore the feasibility of real-time spatial video generation for both Vidu S2-Avatar and Vidu S2-Editing. Compared with Vidu S1, Vidu S2-Avatar supports real-time 720p video generation, generation with dynamic references that can be updated at any moment, and stronger instruction following, such as dancing. Vidu S2-Editing supports editing a video stream in real time, including style rendering, clothing replacement, character replacement, and background replacement. Experiments show that Vidu S2 outperforms all baselines. A playable online demo is available at https://vidu.com/vidu-stream.