Vision encoder layers with certain weight traits improve fine-tuning results
Pre-PEFT Probing: Weight Statistics and Perturbation Robustness for Layer Selection in VLM Vision Encoders
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Fine-tuning large vision-language computer models usually involves adjusting many parts at once, which can be inefficient. The authors found a way to check specific layers in these models before fine-tuning by looking at their internal weight properties and stability to small changes. These checks can predict which layers will improve performance the most when fine-tuned individually. This approach helps reduce the amount of training needed while still getting good results.
What this means in practice
- •For machine learning engineers: Select specific vision encoder layers to fine-tune using weight statistics to reduce training resources without hurting accuracy.
- •For software developers for computer vision: Improve model adaptation by identifying robust layers before fine-tuning, making updates faster and more efficient.
Authors
Qingtao Xia, Jiahua Bao, Siyao Cheng, Jie Liu
Abstract
We propose a pre-fine-tuning probing method for Parameter-Efficient Fine-Tuning (PEFT) layer selection, aiming to obtain more stable and higher gains with fewer trainable parameters when adapting large vision--language models (VLMs). Unlike the common practice of applying LoRA and other adapters to all layers at once---where layer selection often relies on heuristic rules---we focus on the vision encoder and directly evaluate the "adaptability'' of each Transformer layer. Specifically, we characterize each layer from two perspectives: (i) the statistical properties of its Q/K/V projection weights (e.g., norms and condition numbers); (ii) robustness under controlled parameter perturbations. We then systematically compare these indicators with the downstream performance gains brought by applying PEFT to a single layer. Across experiments covering seven benchmarks and five PEFT variants, we observe a consistent correlation: layers (or matrices) with larger weight norms and higher condition numbers are usually more robust to perturbations and are more likely to yield larger fine-tuning gains. These results show that distribution-statistics analysis and perturbation tests before fine-tuning can provide practical signals for adaptation-layer selection, thereby maintaining or improving performance while reducing trainable parameters.