LynnReal-Omni enables controllable multimodal video generation with real-time speed
LynnReal-Omni: Native multi-modal Video Generation for Agentic Visual Workflows
Computer Vision and Pattern Recognition
Summary
Generating videos from descriptions or images is tricky because results can vary and be hard to control. The authors created LynnReal-Omni, a system that combines different inputs like text, images, 3D scenes, and game footage to produce videos that are more accurate and consistent. They built powerful models that work together to handle many video tasks, like editing, restoring, and creating long sequences. Their system can generate short videos quickly enough for streaming, making it easier to create complex videos with precise control.
What this means in practice
- •For video content creators: Enable fast and precise video generation from mixed inputs like text, images, and 3D scenes for streamlined video production workflows.$Commercial implications: Supports products that offer quick, customizable video creation tools for filmmakers and digital media producers.
- •For game developers: Automatically generate or edit videos using gameplay recordings and 3D render inputs to create trailers or cutscenes with consistent visuals.
Authors
Xiaofeng Mao, Peijia Lin, Shaohao Rui, Yibo Zhang, Haibin Wan, Weijie Ma
Abstract
Video diffusion models are stochastic and hard to control: precise content often requires repeated sampling without guaranteed success, and long-horizon scenes drift in appearance, interactions, and temporal coherence. Agentic visual creation provides explicit references, editable 3D scenes, or executable game states for stable control, but does not by itself guarantee high object or character fidelity. Combining the two can enable stable, high-quality generation. To realize this combination, we present LynnReal-Omni, a native multimodal video generation framework built on a 32B shared multimodal diffusion transformer that unifies text-to-video, image-conditioned generation, reference-guided generation, structural control, editing, degraded video restoration, and long-video generation. It accepts heterogeneous visual inputs, including appearance references, editable 3D renders, and game recordings, allowing agents to compose visual conditions within a unified model. We also train a dedicated 27B Flash shared multimodal diffusion transformer for real-time rendering. We build a systematic data pipeline for video cleaning, subject association, multimodal annotation, and aligned control construction, yielding a curated corpus of multi-shot audiovisual segments, and introduce MSAVP, a 100-prompt, 20-metric evaluation design that separates instruction following, generating plausibility, visual quality, temporal behavior, and audio coordination. LynnReal-Omni-Flash further reduces inference cost through model and decoding acceleration, including a lightweight VAE decoder; on one H100, warm generation and decoding of a 22-frame 540p video take 843 ms with LynnReal-Omni and 377 ms with Flash. These results provide a foundation for real-time streaming video generation, making LynnReal-Omni a unified, controllable, and efficient basis for agentic visual creation.