VIBE: Video Instruction-aligned Background music gEneration
2026-08-31 • Sound
SoundArtificial IntelligenceComputation and LanguageComputer Vision and Pattern RecognitionMachine Learning
AI summaryⓘ
The authors created VIBE, a new model that makes music from both videos and text instructions. Unlike older models, VIBE uses a special way to connect planning and refining steps, and it learns through rewards that judge both clear rules (like matching the tempo) and subjective qualities (like how well the music fits the video). Tests show that VIBE follows instructions better and offers more control while still producing good quality music. This approach helps overcome previous problems with controlling the music generation based on video and text cues.
video-to-music generationdiffusion autoregressive modelscross-modal conditioningreward modelingaudio-visual alignmenttempokeymultimodal learninginstruction followinggeneration fidelity
Authors
Aryan Vijay Bhosale, Vaibhavi Lokegaonkar, Vishnu Raj, Gouthaman KV, Sreyan Ghosh, Ramani Duraiswami, Lie Lu, Dinesh Manocha
Abstract
Current video-to-music (V2M) models lack semantic control and fail to penalize instruction violations, largely due to their reliance on reconstruction objectives and the representational bottleneck of static cross-modal conditioning in Diffusion Autoregressive (DAR) architectures. To resolve this, we introduce VIBE, a novel text-and-video-to-music (T+V2M) generation model that leverages: (1) Conditioning Connection, a depth-wise cross-layer conditioning mechanism that dynamically bridges the planning and diffusion refinement heads and (2) a comprehensive reward modeling taxonomy, optimizing for both hard, verifiable constraints (e.g., tempo, key) and soft, subjective qualities (e.g., musicality, multimodal alignment) with a structured 5-stage training curriculum. Upon evaluation using audio-visual alignment, instruction following, and audio quality metrics, along with a subjective human evaluation study, we observe that VIBE demonstrates enhanced controllability and instruction adherence while performing comparably to most evaluated baselines on generation fidelity and multimodal alignment.