JointREPA improves audio and video matching in underspecified prompts
Beyond Co-Generation: Rethinking Jointness in Joint Audio-Video Generation
Computer Vision and Pattern Recognition
Summary
Generating videos with matching sounds based on text descriptions is tricky when the description is vague. The authors studied how current AI models struggle to truly link the sounds and visuals when details are missing from the text. They introduced a new way to measure this 'jointness' and created a method called JointREPA to help improve it. Their new benchmark, UNSAIDBench, tests how well models can match audio and video when the text doesn’t specify everything. Results show JointREPA helps models make sounds and images that fit together better.
What this means in practice
- •For multimedia app developers: Create apps that generate synchronized video and audio from brief or vague text descriptions for enhanced realism.
- •For digital content producers: Improve the automatic matching of sound and visuals when producing multimedia content with loosely defined scripts or concepts.
Authors
Jihoo Jung, Ji-Hoon Kim, Suyeon Lee, Gia Thinh Pham, Joon Son Chung
Abstract
Our aim is to advance the understanding of genuine audio-visual coordination in joint text-to-audio-video (T2AV) generation, with particular emphasis on semantic consistency. Evaluating audio-visual consistency in T2AV generation is nontrivial, as detailed text prompts obscure whether such consistency stems from genuine cross-modal coordination or merely from independent text following. To isolate the former, we introduce intrinsic jointness: the ability of audio and video to produce the same realization for details left underspecified by the prompt. Using semantically underspecified prompts (e.g., "musical instrument playing" rather than "electric guitar playing"), we investigate four recent models and reveal their limited intrinsic jointness. To address this limitation, we propose JointREPA, a simple representation alignment-based post-training method to facilitate cross-modal coordination. Since existing benchmarks use semantically detailed prompts, we introduce UNSAIDBench, a benchmark designed to evaluate intrinsic jointness under semantic underspecification. Experiments across two distinct models and three benchmarks, including UNSAIDBench, demonstrate that JointREPA consistently improves semantic consistency.