Grounded descriptions improve recognizing everyone in multi-person videos
Grounded Human-Attributed Description and Activity Recognition in Videos
Computer Vision and Pattern Recognition
Summary
Recognizing what many people are doing in a busy video is tough, especially when the activities and details are varied and open-ended. The authors introduce a new task to describe each person’s characteristics and actions in videos, using their exact location to better understand them. They created a new dataset with detailed captions for many people doing different activities in the same video. To do this better, they designed a new training method that helps the model focus on the right person. They also suggest a fresh way to evaluate how well these models work by comparing the video directly with the model's descriptions.
What this means in practice
- •For security operators: Automatically generate detailed descriptions of every person’s actions in surveillance footage to enhance monitoring accuracy.
- •For video content analysts: Produce precise activity and attribute descriptions for each individual in complex group scenes to improve video indexing and summarization.
Authors
Jay N. Paranjape, Jurijs Nazarovs, Rohan Sarkar, Yue Guo, Lipeng Ke, Sankar Venkataraman, Vishal M. Patel, Nataraj Jammalamadaka
Abstract
Understanding characteristics and activities of individuals in complex multi-person environments is crucial for real-world applications. However, existing datasets often simplify this problem to grounded group activity recognition, single-person activity recognition, or close-set activity recognition, thus limiting the generalizability of models trained on these datasets. In this work, we introduce the task of Grounded Human-Attributed Description and Activity Recognition (GHADAR), which involves describing the characteristics and activity descriptions of \textbf{every} person in a video, provided the location of the person in the video, which is more practical for real-world applications. To facilitate this, we introduce a new dataset derived from AVA-Actions by generating open-set captions for the person description and activity. In addition, we propose a novel method to effectively utilize the information contained in grounding during training by constraining the cross-attention masks during training in VLMs to improve performance for this task. Our experiments show that our method outperforms SOTA VLMs on this task. Finally, we demonstrate the limitations of existing evaluation metrics, which are overly reliant on human-annotations and exact text-text matching. As an added video-based evaluation, we propose a holistic VLM-based evaluation schema that compares concepts {\em directly} between the video and the generated predictions. Thus, in this work, we develop a complete framework for GHADAR, including a dataset, a novel method and an evaluation schema, thereby establishing a strong foundation for future research in this domain.