New benchmark reveals safety risks in text to audio video generation
AV-SafetyBench: A Safety Benchmark for Text-to-Audio-Video Generation
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Some AI systems can create videos, speech, and sounds from just a written description. This makes checking for unsafe content trickier because dangers might come from the sound, the images, or how they work together. The authors introduce AV-SafetyBench, a tool that tests these systems by looking at video and audio separately and together. They found many unsafe outputs and showed that checking video alone misses a lot of problems. Their work helps better understand and improve the safety of these complex AI-generated multimedia creations.
text-to-audio-video generationsafety benchmarkmodalitiesaudio-visual contentrisk assessmentcross-modal interactionunsafe contentmachine learningmultimedia AIcontent evaluation
Authors
Suah Choi, Tae-Young Lee, Gyeong-Moon Park
Abstract
Recent text-to-audio-video (T2AV) models jointly generate video, speech, sound effects, and ambience from a single text prompt. This capability poses new challenges for safety evaluation, as unsafe content may be conveyed through the audio track or arise only when the visual and audio tracks are interpreted jointly. Existing safety benchmarks largely focus on either generated video or generated audio in isolation and are therefore not designed to capture these risks. To close this gap, we introduce AV-SafetyBench, the first safety benchmark developed specifically for T2AV generation. AV-SafetyBench comprises a four-axis, 13-category taxonomy and 5,200 manually reviewed prompts that specify visual scenes, speech, and non-speech audio. Our evaluation protocol assesses each output under three views: Full-AV, Video-Only, and Audio-Only. It then uses the Video-Only and Audio-Only judgments to assign Full-AV unsafe outputs to one of four risk sources: Video-Only, Audio-Only, AV-Both, or AV-Joint. We evaluate five open-source T2AV models and validate the automated Full-AV judgments against human annotations. Across the five models, Full-AV Unsafe Rates range from 25.1% to 49.4%. Beyond these aggregate rates, risk-source analysis reveals that, for four of the five models, Audio-Only and AV-Joint cases-unsafe outputs missed by video-only evaluation-account for 41.6-48.3% of Full-AV unsafe outputs for which a risk source could be assigned. In the Cross-Modal Harm Emergence category, AV-Joint accounts for 87.5% of unsafe outputs withan assigned risk source. Together, these findings demonstrate the value of AV-SafetyBench for evaluating T2AV safety across the visual and audio modalities and their interaction.