Vision foundation models performance compared under efficient finetuning methods
A note on data-efficient finetuning of vision foundation models
Computer Vision and Pattern RecognitionMachine Learning
Summary
It is common to adapt big pre-trained image models to new tasks by fine-tuning them. Visual Prompting (VP) is a newer method for this that promises to do it efficiently. The authors compare VP and other fine-tuning methods across different types of vision models, including a new family called Mamba. They also test how adding regularization techniques affects VP's performance. This helps users understand which methods work best given limited computing resources.
What this means in practice
- •For machine learning engineers: Optimize transfer learning strategies by selecting fine-tuning methods and regularizations that improve performance for new vision tasks on limited data and compute.
- •For ai product teams: Deploy pre-trained vision foundation models with efficient adaptation methods like visual prompting, reducing the cost and time of customizing AI for specific applications.
Authors
Niclas Hergenröther, Antonio Orvieto
Abstract
Adapting foundation models to a new task within or beyond the training domain is the de facto standard for building specialized machine learning models. Recently, Visual Prompting (VP) has emerged as a promising technique for efficient knowledge transfer. As new foundation model families~(like Mamba) get introduced and VP pipelines such as AutoVP reach greater maturity, we find a growing need for a systematic evaluation of current approaches. In this work, we assess the performance of the latest models, comparing them to earlier architectures and alternative fine-tuning methods, to better understand the progress, challenges and opportunities in the field of efficient fine-tuning under resource limitations. Towards this goal, this paper provides a concise empirical overview of the interactions among foundation model families (Attention-, Convolution-, and Mamba-based) and transfer paradigms: VP, Linear Probing (LP), Full Fine-tuning (FFT) and Low Rank Adaptation (LoRA). Our work builds up on previous findings by broadening the selection of evaluated models and techniques while tuning hyperparameters. In the interest of delivering practical guidelines for the user, we also explore application of prevalent regularization techniques to boost performance in the context of VP.