Multilingual raw video dataset supports event search and summary
MultiVENT-Raw: A Benchmark for Retrieval and Reasoning over Raw Videos
Computer Vision and Pattern RecognitionInformation Retrieval
Summary
Videos taken directly from phones or security cameras often lack helpful titles or captions, making it hard for computers to understand or find key info. The authors created MultiVENT-Raw, a large collection of raw videos paired with specific event descriptions and questions about those events. They included human reviews of which videos are relevant and key facts from those videos to help train and test computer models. Their tests show that even advanced AI systems find it difficult to accurately retrieve and summarize videos in this raw format.
What this means in practice
- •For social media platforms: Improve automatic search and summary of unedited user-uploaded videos based on specific events, aiding content discovery.
- •For security operations teams: Enhance event detection and reporting from large collections of raw CCTV footage without relying on metadata or captions.
Authors
Reno Kriz, David Etter, Alexander Martin, Cameron Carpenter, Debashish Chakraborty, Hannah Recknor, Reihaneh Iranmanesh, Matthew Maciejewski, Kenton Murray, Eugene Yang, Benjamin Van Durme, Aaron Steven White, Andrew Yates, William Walden
Abstract
Online information is increasingly consumed in video format. Much of this comes in the form of *raw video*: continuous footage taken on a cell phone, with a hand-held camera, or via CCTV, which is then directly uploaded to social media platforms and content sharing services. Whereas professional or even amateur-edited footage tends to feature scripted speech, chyrons, graphics, and metadata that help contextualize its subject matter, raw video typically contains none of these things, making it a much more challenging medium for information retrieval and machine understanding. To facilitate progress in this domain, we release MultiVENT-Raw, a multilingual collection of nearly 120,000 primarily raw videos (over 5,300 total hours), paired with 130 events and 222 event-centric queries, along with human-annotated video relevance judgments and human-extracted key facts for relevant videos. MultiVENT-Raw supports both a retrieval task---to identify videos in the collection relevant to a query event---and a generation task---to summarize event-related videos into a coherent report for a target user. We benchmark strong baselines on MultiVENT-Raw, showing both tasks to be challenging even for some of the latest multimodal models.