Marine video dataset advances motion aware understanding tasks
MarineLife-16K: An Attribute-Rich Video-Text Dataset for Motion-Aware Marine Video Understanding
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Understanding videos of underwater life is important but challenging because existing collections are small and lack detailed descriptions. The authors created a big new dataset with 2,000 underwater videos, each paired with detailed text about objects, their looks, movements, and the environment. They also built many questions about the videos to help train and test AI models. They tested different video language models and found they understand moving objects well but have trouble with complex camera movements and reasoning when the camera is still.
What this means in practice
- •For marine biologists: Use the dataset to build AI tools that automatically analyze underwater videos for species behavior and environment monitoring.
- •For video ai developers: Develop and benchmark motion-aware video understanding models using richly annotated marine video data to improve object and camera motion recognition.
- •For environmental monitoring companies: Create automatic monitoring systems for marine environments using the dataset to detect complex motions and contextual changes in underwater videos.$Commercial implications: Enables selling AI-powered marine environment monitoring services that detect objects and motions from video feeds.
Authors
Quang-Trung Truong, Thanh-Nghia Nguyen, Phu-Phong Vo, Jaeyeon Kim, Cheng Chen, Jie Zhang, Binh-Son Hua, Duc Thanh Nguyen, Ivor Tsang, Sai-Kit Yeung
Abstract
Marine video understanding is crucial for a wide range of real-world applications. However, existing marine video datasets suffer from limitations, including the limited number of data samples, fine-grained text descriptions, and detailed visual attributes. To address these limitations, we introduce a large-scale, video-text dataset featuring rich, in-the-wild visual attributes tailored for complex marine environments. Our dataset comprises 2,000 video-text pairs annotated with 13 attributes covering up to four objects per scene, object visual/geometric properties, behaviors, environmental context, and camera motion, spanning diverse lighting conditions, times of day, and 7 countries. From these annotations, we generate 16,080 video-question pairs (12,080 multiple-choice, 4,000 open-ended). Furthermore, we benchmark both proprietary and open-source video-LLM models on video question answering and video captioning tasks. We find that (i) proprietary models often achieves superior performances; (ii) VLMs handle object motion well but struggle with complex camera motion; (iii) closed-source models struggle to perceive camera motion; and (iv) VLMs struggle to reason in static cameras.