Distributed subliminal learning shares knowledge without sending model updates
Distributed Subliminal Learning: Replacing Model Updates with Random-Carrier Outputs
Machine LearningArtificial Intelligence
Summary
Collaborative AI training usually requires sharing parts of models, which can be large and cause conflicts. The authors propose a new way called distributed subliminal learning, where participants send only how their models behave on unrelated random inputs instead of actual updates. A central coordinator uses these responses to create a shared model, reducing communication needs and avoiding direct weight conflicts. They tested this method on language and image tasks and found it achieves good results while sending much less data.
What this means in practice
- •For federated learning engineers: Cut communication costs by sharing random input responses instead of model weights in collaborative AI training setups.
- •For llm deployment teams: Combine independently adapted large language models more efficiently via behavioral output sharing to improve performance and reduce upload size.
Authors
Dario Fenoglio, Gabriele Dominici, Martin Gjoreski, Marc Langheinrich
Abstract
Collaborative learning typically exchanges model parameters: federated clients communicate updates, while independently adapted foundation models are combined by exchanging adapters or checkpoints. This makes communication scale with model size and requires local specializations to be reconciled in weight space, where interference is common. We ask whether knowledge can instead be shared through model behavior on task-unrelated inputs. We introduce Distributed Subliminal Learning (DSL), a collaborative learning primitive in which participants adapt a common model locally, probe it with task-unrelated inputs, and transmit only the resulting carrier outputs. A coordinator pools these outputs and distills them into a shared model. The primitive supports one-shot foundation-model composition through carrier completions and iterative federated learning through carrier logits, without transmitting model updates or requiring task-related proxy data. In LLM composition, compared with LoRA averaging, DSL achieves higher preference retention (94.56% vs. 87.76%) and a larger GSM8K gain over the base model (22.0 vs. 0.6 points), while reducing upload by 30.6-49.0$\times$. In federated classification, DSL reaches 96.83% on MNIST with 8.9$\times$ less uplink than FedAvg and provides lower-communication operating points on CIFAR-10 and Tiny ImageNet. These results establish random-carrier outputs as a practical communication primitive for knowledge sharing across distinct collaborative learning paradigms.