Self supervised model tuning improves task learning efficiency
A Light Bilevel Refinement Aligns Self-Supervised Representations for Stronger Task-Specific Learning
Machine LearningArtificial IntelligenceComputer Vision and Pattern Recognition
Summary
Training artificial intelligence models to do different jobs often starts with teaching them in a way that doesn’t rely on human labels, called self-supervised learning. But directly adjusting these models for a specific job can be tricky since their initial training goals don’t perfectly match the new task, which can reduce their usefulness. The authors propose BiSSLight, a faster and more efficient method that gently refines the model’s knowledge between these stages. This helps the models perform better on various tasks, especially as their size grows, and it saves a lot of computing time.
What this means in practice
- •For machine learning engineers: Speed up and improve the fine-tuning process of large pretrained models for specific tasks using efficient bilevel refinement.
- •For computer vision teams: Enhance the performance of vision transformers on diverse image-related tasks with a scalable intermediate training step.
Authors
Gustav Wagner Zakarias, Zheng-Hua Tan
Abstract
Self-supervised pretraining learns representations that are broadly transferable across downstream tasks, yet direct fine-tuning can be suboptimal due to misalignment between self-supervised and downstream task objectives, potentially degrading pretrained features beneficial to the downstream task. The BiSSL framework addressed this by introducing a transitional training stage formulated as a bilevel optimization problem, in which the downstream task objective guides the self-supervised learning process in refining pretrained representations to better facilitate subsequent fine-tuning. However, BiSSL relies on conventional bilevel optimization solving techniques whose costly implicit hypergradient approximations render the method increasingly impractical for contemporary model architectures. To make it efficient and scalable, we introduce BiSSLight, which combines M-FAC-based implicit gradient approximation with parameter-efficient fine-tuning via LoRA, enabling efficient application at larger scales that were previously impractical. Evaluation across multiple downstream tasks and contemporary model architectures shows that BiSSLight consistently improves downstream performance, with gains becoming more pronounced as model size increases despite stronger baselines. The method is highly computationally efficient, reducing computation time by more than a factor of ten compared to its predecessor on a ViT-H backbone.