Zero shot video highlight detection uses text and synthetic images
Zero-shot video highlight detection based on text descriptions and synthetic images
Computer Vision and Pattern Recognition
Summary
Finding the most interesting parts of a video is useful for summaries and recommendations but typically needs lots of labeled examples. The authors created a new method that works without any training on highlight examples. Their method uses a language model to guess what exciting moments might be in a video based on simple info like a title. Then it makes fake images of those moments and compares them to actual video frames using a tool that links text and images to spot highlights. This way, they can find important scenes in videos without needing extra training data.
What this means in practice
- •For video platform engineers: Automatically detect key moments in videos based only on metadata, improving video summaries and recommendations without labeled highlight data.$Commercial implications: Enables scalable highlight detection for video services without costly annotation, supporting commercial content curation and personalized viewing.
- •For advertising teams: Identify engaging video clips from large catalogs using only titles or categories, to select prime content for ad placement.
Authors
Michal Byra, Alberto Presta, Grzegorz Stefanski, Krzysztof Arendt
Abstract
Detecting video highlights, the most informative or engaging moments in a video, is important for applications such as video summarization and content recommendation. We propose a zero-shot framework that combines CLIP, large language models (LLMs), and diffusion models. Given lightweight video metadata, such as a title or category, an LLM generates textual descriptions of likely highlight events. These descriptions are further converted into synthetic visual prototypes using a diffusion model. Textual and visual representations are matched to video frames using CLIP, enabling frame-level highlight detection without highlight annotations or dataset-specific training. Experiments on TVSum and SumMe demonstrate strong zero-shot performance, with particularly favorable results on TVSum. The proposed approach provides an effective framework for metadata-conditioned zero-shot video highlight detection.