Summary
Blind and low-vision people rely on audio descriptions to understand videos, but these descriptions need to fit into natural pauses in speech to avoid confusion. The authors created a high-quality dataset called MTG-AD with videos and timed audio descriptions, made by paired blind and sighted professionals. They also developed a way to measure how well AI models generate these timed descriptions without overlapping existing speech. Their tests show that while AI can describe videos well and reduce speech overlaps, some timing and flow issues remain. Users slightly prefer human-made descriptions but find AI-generated audio descriptions very helpful for understanding videos.
What this means in practice
- •For video platform developers: Create automated audio descriptions that fit into silent gaps of user videos to enhance accessibility for blind viewers.
- •For media production teams: Use the MTG-AD dataset and evaluation tools to improve timing and quality of inline audio descriptions for diverse video content.
Authors
Subhashini Venugopalan, Jenny Tan, Taylor Roper, Jimmy Tobin, Anton Kast, Alicia Martin, Sam Sepah, Amy Pavel
Abstract
Audio descriptions (AD) are essential for making visual media accessible to blind and low-vision (BLV) users. While Multimodal Large Language Models (MLLMs) offer a scalable solution for generating audio descriptions on-demand, their performance on ``inline'' audio descriptions -- which must fit within existing silences in a video -- remains under-explored, particularly for diverse user-generated content (UGC). We present a comprehensive investigation into MLLM-generated inline audio descriptions. First, we introduce MTG-AD, a professionally annotated dataset of 36 hours of videos, each averaging $\sim$5 mins. and 8k+ descriptions in total, co-created by paired blind and sighted AD professionals to provide a high-quality benchmark for this task. Second, we propose MTG-AD-Eval, an evaluation framework based on audio description expert guidelines that provides fine-grained, reference-free actionable metrics for model improvement. Crucially, our framework evaluates the end-to-end task: it renders the generated scripts into audio tracks to assess their fit within the original video's natural silences, minimizing disruptive overlaps. Our analysis across frontier MLLMs reveals that while models accurately describe visual events and substantially reduce speech overlaps, gaps persist in narrative pacing and flow. Finally, we empirically validate that our automated and reference-free metrics correlate significantly with human quality ratings from sighted raters and 7 BLV participants, finding that while users narrowly prefer human-authored audio descriptions, they find MLLM-generated audio descriptions highly beneficial for comprehension. We provide actionable system design guidance and release MTG-AD at https://hf.co/datasets/google/mind-the-gap-inline-ad and MTG-AD-Eval at https://github.com/google/mtg-ad.