Fine tuning large language models improves driving video captioning quality
Comparative study of adapting pre-trained models for driving behavior video captioning
Computer Vision and Pattern RecognitionMachine Learning
Summary
Understanding what is happening in driving videos is important for autonomous vehicles to explain their actions. This paper looks at different ways to adapt large language models that normally understand text to describe driving scenes from video data. The authors tested methods like full fine tuning, low-rank adaptation, and prompt engineering on specific models and datasets. They found that full fine tuning of their main model performed well and even beat some baseline results on certain metrics. They also explored limitations of other approaches like prompt engineering on another model.
What this means in practice
- •For autonomous vehicle software teams: Use fine tuned language models to generate descriptive explanations of driving behavior from video data when deploying self-driving cars.
- •For video data labeling teams: Deploy adapted language models to assist with annotating driving videos by automatically describing driving situations from raw footage.
Tested on one dataset.
Authors
Sayak Mallick, Philipp Geiger, Augustin Kelava
Abstract
This report examines and compares some of the many fine tuning and prompting methods existing, applying them within the domain of autonomous driving. The idea is to compare these methods by adapting a Large Language Model (LLM) on a video dataset. LLM's have become extremely good at achieving a good understanding of different forms of data and this study aims to induce a low dimensional understanding of driving situations into our primary test model SpaceTimeGPT. Experiments on BDD-X (Berkeley DeepDrive eXplanation) dataset demonstrate good performance of the full fine tuning framework on some automatic metrics, and in some metrics, it even surpasses the baseline. We also try Low-Rank Adaptation (LoRA) and prompt engineering on VideoLLaVA model and discuss its limitations.