Prompt guided vision language models improve video recognition accuracy
Revisiting Natural Language Prompts from Large Language Models for Open-Vocabulary Video Recognition
Computer Vision and Pattern Recognition
Summary
Video recognition models that understand language struggle to recognize many types of videos well. The authors show that creating better text descriptions using large language models can help these vision-language models work better. They introduce a method called P2CLIP that makes categories clearer and improves how video details match text descriptions. This method also uses knowledge from language models to better process the timing of video frames. Tests show P2CLIP improves recognition especially for new or limited data categories.
What this means in practice
- •For video software engineers: Improve video classification software to recognize new or rare categories without extensive retraining, using prompt generation and alignment improvements.
- •For social media content teams: Automatically tag diverse video content with more accurate labels by enhancing video understanding with language-guided prompts.
Authors
Haichen He, Weibin Liu, Weiwei Xing
Abstract
Vision-language models (VLMs) like CLIP have shown great success in image tasks, however, applying CLIP to open-vocabulary video recognition poses challenges. Due to the limited text information, directly fine-tuning CLIP on video recognition datasets may lead to overfitting and hinder its generalizability. A prevalent method is to enrich text information by generating prompts using Large Language Models (LLMs). However, limited efforts have been made to explore how to generate higher-quality prompts that can contribute to classify unseen categories, and how to further enhance vision-text alignment based on these prompts. In this paper, we propose P2CLIP (Prompt-guided CLIP), which aims to bridge this gap by exploring the impact of prompts across three aspects: category, video, and frame. At the category level, we argue that LLM-generated prompts under simple instructions may suffer from semantic overgeneralization, which can blur the category boundaries and hinder the performance of the model. To address this issue, we design Generic-Discriminative Prompt using LLMs to obtain discriminative category embeddings with rich semantic information. At the video and frame levels, we first use VLM and LLM to generate frame and video descriptions for enhancing vision-text alignment. Then we propose a Knowledge-Guided Temporal Aggregation mechanism, which leverages the expert knowledge derived from LLMs to guide the temporal modeling of videos and encourage a better construction of the joint visual-textual feature space. Extensive experiments across various benchmarks, including zero-shot and few-shot settings, demonstrate the effectiveness of our method.