Papers for

video data labeling teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Fine tuning large language models improves driving video captioning quality

Comparative study of adapting pre-trained models for driving behavior video captioning

Abstract: This report examines and compares some of the many fine tuning and prompting methods existing, applying them within the domain of autonomous driving. The idea is to compare these methods by adapting a Large Language Model (LLM) on a video dataset. LLM's have become extremely good at achieving a good understanding of different forms of data and this study aims to induce a low dimensional understanding of driving situations into our primary test model SpaceTimeGPT. Experiments on BDD-X (Berkeley DeepDrive eXplanation) dataset demonstrate good performance of the full fine tuning framework on some automatic metrics, and in some metrics, it even surpasses the baseline. We also try Low-Rank Adaptation (LoRA) and prompt engineering on VideoLLaVA model and discuss its limitations.

Wed 30 SeptComputer Vision and Pattern RecognitionMachine Learning

The gist

Understanding what is happening in driving videos is important for autonomous vehicles to explain their actions. This paper looks at different ways to adapt large language models that normally understand text to describe driving scenes from video data. The authors tested methods like full fine tuning, low-rank adaptation, and prompt engineering on specific models and datasets. They found that full fine tuning of their main model performed well and even beat some baseline results on certain metrics. They also explored limitations of other approaches like prompt engineering on another model.

Open → 2609.39542v1