OmniXRef enables video editing with text image and video references
OmniXRef: A Unified Framework for Multimodal Multi-Reference Video Editing
Computer Vision and Pattern Recognition
Summary
Video editing often involves giving instructions using text, images, or videos to change how a video looks or moves. The authors created OmniXRef, a system that lets users combine all these types of instructions together to edit videos in flexible ways. They built tools to handle many references at once and trained their model with specially made data to handle complex edits. Tests show this approach works better than others focused on single types of edits.
What this means in practice
- •For video production teams: Edit videos by combining text, images, and video examples to control appearance, motion, style, and effects in one tool.$Commercial implications: This makes advanced video editing products possible that allow multi-reference inputs for more precise and flexible creative control.
- •For graphic design studios: Use multimodal references to automate complex style and visual effect transfers across video projects.
Authors
Xiaoyan Cong, Tianyu Wang, Hyeonho Jeong, Zhifei Zhang, Yuqian Zhou, Yijun Li, Shaoteng Liu, Zhe Lin, Srinath Sridhar, Soo Ye Kim
Abstract
Creative intent in video editing can be expressed through text instructions (e.g., "remove the people"), image references (e.g., subject appearance), video references (e.g., camera movement), or a combination of the above. We introduce OmniXRef, a unified framework for multimodal multi-reference video editing that allows users to combine variable numbers of text, image, and video references to control diverse attributes of a source video. We first devise a unified input design with a reference-bank interface and introduce $R^2$-RoPE to support generalization to more references. Then our model, OmniXRef-Model, uses a single-stream diffusion transformer to jointly process the source video and multimodal context from an MLLM in an interleaved sequence. To address the scarcity of training data for video-reference editing, we construct OmniXRef-Data using synthesis pipelines for different editing tasks and combinations of references. We use curriculum learning to progressively increase task complexity and resolution. To support systematic evaluation of video as a control modality, we introduce OmniXRef-Bench, a benchmark for video-reference attribute transfer across Camera, Motion, Content, Style, and Visual Effects (VFX). Comprehensive evaluations show that our OmniXRef-Model outperforms task-specific baselines and commercial models.