AutoEncoder-Compressed Parallel Split Learning for Pre-trained Model Fine-Tuning
2026-07-20 • Distributed, Parallel, and Cluster Computing
Distributed, Parallel, and Cluster ComputingMachine Learning
AI summaryⓘ
The authors address challenges in training large AI models on devices with limited computing power by splitting the model's work between the device and a server. They note that existing methods to reduce communication between device and server either use simple shortcuts or need to be trained alongside the model, which causes problems. To solve this, they propose AE-PSL, a method that uses a small autoencoder to compress data exchanged during training without disrupting the pre-trained model. Their approach includes a special alignment step to make sure the compression works well with the existing model features before fine-tuning begins.
Distributed Fine-TuningFoundation ModelsEdge DevicesParallel Split LearningAutoEncoderIntermediate ActivationsGradientsFeature DistributionCommunication Compression
Authors
Bas Meuwissen, Vasileios Tsouvalas, Nirvana Meratnia
Abstract
Distributed Fine-Tuning (DFT) of large-scale Foundation Models (FMs) on resource-constrained edge devices is limited by local compute constraints and communication overhead. Parallel Split Learning (PSL) reduces client-side computation by keeping few model layers on each client and offloading the remaining computation to the server; however, clients must exchange intermediate activations and gradients with the server at every training step. Existing SL communication-compression methods mainly rely on task-agnostic heuristics, such as sparsification and quantization. While learnable SL compressors can better adapt to intermediate representations, they require co-training with the target model. Therefore, directly inserting them into off-the-shelf FMs introduces feature-distribution misalignment and degrades DFT performance. To address this, we propose AE-PSL, a communication-efficient PSL framework that compresses intermediate activations and gradients using a lightweight AutoEncoder (AE) placed at the split layer. To ensure compatibility of AE compression with pre-trained FMs, AE-PSL introduces a novel two-stage alignment mechanism, which adapts the AE to the pre-trained model's feature manifold and client-specific feature distributions before DFT.