Video guides better audio and motion in multimodal generation
All modalities are equal, but video is more equal: Closing the Cross-Attention Gap in Joint Video Generation
Computer Vision and Pattern Recognition
Summary
Videos show many parts of an event, like how things look and move over time, while other data types like sound or body motion show only some parts. The authors found that when teaching AI models to create combined videos and sounds or movements, the connection from video to the other data works well, but the other way around is weak. They created a new technique, RecCAR, to make the weaker connection stronger by aligning it with the stronger one. This improves how well the AI can generate synced and realistic videos and sounds or body movements together.
What this means in practice
- •For video content creators: Generate more accurate videos that align better with accompanying sounds or body motions in multimedia projects.$Commercial implications: This enables production-quality AI-generated videos with synchronized audio and motion for entertainment and advertising industries.
- •For augmented reality developers: Create more realistic AR experiences by improving how virtual videos match user motions and environmental sounds.
Authors
Ohad Rahamim, Dvir Samuel, Idan Schwartz, Gal Chechik
Abstract
Video is a rich representation of a physical event, capturing appearance, geometry, motion, and temporal evolution. Other modalities, such as 3D body motion or audio, encode narrower aspects of the same event. We find that joint multimodal diffusion transformers exhibit a corresponding asymmetry in cross-modal correspondence: companion modalities develop strong correspondences to video, but the reciprocal correspondences through which they constrain video remain substantially weaker. We express both directions as comparable correspondence distributions over video tokens and define their disagreement as the reciprocal correspondence gap. We introduce RecCAR, standing for Reciprocal Cross-modal Attention Regularization, a KL regularizer that uses the well-established video-to-modality correspondence as a fixed reference and aligns the weaker modality-to-video correspondence toward it. Across joint video-motion and video-audio generation, RecCAR improves the Human Anatomy score from 0.69 to 0.75 and reduces audio-video desynchronization from 0.804 to 0.752, while improving overall generation