SOLAR: Communication-Efficient Model Adaptation via Subspace-Oriented Latent Adapter Reparametrization
2026-04-09 • Machine Learning
Machine LearningComputation and LanguageComputer Vision and Pattern Recognition
AI summaryⓘ
The authors introduce SOLAR, a method to shrink the size of updates needed when fine-tuning large AI models, like language or vision models, so they use less memory and are faster to share. SOLAR works by representing fine-tuning updates as combinations of important directions already found in the original model, plus some controlled randomness. This approach keeps the model's performance intact while making the updates smaller and easier to handle, which is especially helpful for devices with limited resources. The authors tested SOLAR on various models and tasks and showed it works well without needing changes to existing fine-tuning methods.
Parameter-Efficient Fine-TuningLoRALow-Rank AdaptersSingular VectorsModel CompressionSubspace SimilarityFoundation ModelsLLaMAGPTViT
Authors
Seyed Mahmoud Sajjadi Mohammadabadi, Xiaolong Ma, Lei Yang, Feng Yan, Junshan Zhang
Abstract
Parameter-efficient fine-tuning (PEFT) methods, such as LoRA, enable scalable adaptation of foundation models by injecting low-rank adapters. However, their communication and storage costs remain a major bottleneck in resource-constrained settings. We propose SOLAR (Subspace-Oriented Latent Adapter Reparameterization), a post-training compression framework that substantially reduces the communication cost (i.e., the number of parameters to transmit or store) of PEFT adapters. SOLAR expresses each PEFT update as a linear combination of basis vectors formed from the foundation model's singular vectors with controlled random perturbations. By exploiting the subspace similarity (the alignment of principal directions) between the foundation model and task-specific fine-tuned updates, SOLAR decouples the adapter size from PEFT structure and ensures compact yet expressive representations. It is model-agnostic and compatible with existing PEFT methods, including LoRA, AdaLoRA, and other adapter modules. We theoretically establish a bound on the reconstruction error. Experiments on language and vision tasks using LLaMA, GPT, and ViT models demonstrate that SOLAR preserves task performance while significantly reducing model representation sizes, offering an effective and communication-efficient solution for deployment in distributed systems and edge devices.