Distributed learning improves when agents share signals on correlated data
Cordial Learning: Distributed Training with Correlated Data
Machine LearningArtificial Intelligence
Summary
When multiple agents or computers try to learn from data, their information can be linked or correlated, especially when they share the same environment. Traditional methods either combine all data centrally, which can risk privacy, or ignore these links, resulting in poorer learning. The authors propose a new way called cordial learning, where agents share simple signals instead of raw data to learn effectively together. They prove this method works well in theory and show it performs strongly on image-based tasks.
What this means in practice
- •For distributed system engineers: Build systems that train AI models where multiple devices have related data but cannot share raw data due to privacy or bandwidth limits.
- •For autonomous vehicle developers: Improve learning from sensor data that is correlated between nearby vehicles without exchanging sensitive raw input data.
Authors
Sarah Shitrit, Ilai Bistritz
Abstract
We consider a distributed learning task with agents that have correlated data. Specifically, the label of an agent depends on the input of other agents for the same sample, and these inputs are also correlated. Correlated data is the reality when agents share the same environment. Existing decentralized methods, such as federated learning, ignore the structure of the problem and perform poorly on correlated data. On the other hand, centralized approaches are infeasible due to privacy and communication constraints. We introduce cordial (correlated and distributed) learning to address this gap by sharing only low-dimensional outputs between the agents while training local models to extract informative signals from peers. This distributed learning induces a game in which the loss function of each agent depends on the models of others. Assuming a linear model, we prove that cordial learning converges with probability one to a globally optimal solution, despite the nonconvex global objective. Experiments on structured multi-digit MNIST tasks demonstrate that cordial learning remains highly effective even in highly nonlinear settings.