Audio video consistency measured at event level improves generation quality

AVC-Reward: Event-Centric Audio-Video Consistency Evaluation

Computer Vision and Pattern RecognitionArtificial Intelligence

Summary

When computer programs create videos with matching sounds, it's hard to check if the sound and video really fit well together, especially for complex scenes with many events happening. The authors designed a new way to check consistency between sound and video by focusing on individual events in the scene rather than the whole clip. They trained a model called AVC-Reward that automatically judges how well audio matches video, and it matches human opinions better than previous methods. This tool can help improve the quality of audio-video generation by guiding the process to make better-matched scenes.

What this means in practice

  • For multimedia content creators: Automatically evaluate and improve audio-video matching in generated scenes to enhance viewer experience.
  • For film editors: Use the event-centric evaluation tool to detect mismatches between audio and video layers for complex multi-event scenes.

Authors

Yuruo Wu, Kaihang Pan, Kai Shen, Jiacheng Li, Shuyuan Tu, Jisheng Dang, Li Tuoyu, Juncheng Li, Siliang Tang

Abstract

As joint audio-video generation models become more capable, they can generate audio-visual scenes exhibiting three forms of complexity: multi-event composition, multi-layer audio mixture, and asymmetric cross-modal relations. However, current evaluations largely reuse metrics from earlier simple and unidirectional tasks. They typically evaluate an audio-video clip as a whole and rely on simple predefined relations, making them unsuitable for these increasingly complex scenes. We propose an event-centric evaluation framework that disentangles audio and visual events at the event level, analyzes each event independently, cross-validates relations across modalities, and aggregates the judgments into a coarse-to-fine, three-level consistency metric. The metric assesses global narrative harmony, event-level temporal synchronization, and fine-grained attribute alignment, providing a general formulation for audio-video consistency evaluation across diverse generation settings and cross-modal relations. We instantiate the metric as AVC-Reward, a trainable audio-video reward model. To evaluate the agreement between automatic judgments and human perception across generation paradigms and relation complexities, we construct AVC-Benchmark, which is primarily built from diverse positive and negative samples generated by various types of audio-video generation models. AVC-Reward achieves state-of-the-art alignment with human judgment among automatic methods, provides interpretable rationales that localize failures, and improves the audio-video consistency of a joint generation model when used as an online reinforcement-learning signal. Project page: https://tsuki0512.github.io/AVC-Reward.github.io/