Streaming federated learning models generalize differently with dependent data
On the Generalization Gap of Streaming Federated Learning with Markovian Data
Machine LearningInformation Theory
Summary
Training shared models across many devices from data arriving over time is called streaming federated learning. When the data points are not independent but follow patterns that depend on previous data (called Markovian dependence), it is unclear how well the model will do on new data. The authors study this question and find that the timing of communication between devices and the amount of such temporal dependence can affect how well the model generalizes. They also show that sharing data across devices may sometimes help more than a single central learner.
What this means in practice
- •For machine learning engineers: Design federated learning systems that balance communication frequency to improve model generalization on temporally dependent streaming data.
- •For time-series forecasting teams: Improve forecasting models by incorporating client collaboration insights from federated learning with Markovian time-dependent data.
Authors
Masoud Kavian, Milad Sefidgaran, Youssef Attia El Hili, Malik Tiomoko, Abdellatif Zaidi
Abstract
Streaming federated learning (SFL) allows clients to train a shared model from continuously arriving data without retaining large local datasets. Such streams often exhibit temporal dependence, yet its effect on generalization remains poorly understood. In this paper we study the expected generalization gap of the aggregated model when client data follow Markov processes, using information-theoretic and rate-distortion techniques. Our framework accommodates heterogeneous Markovian dynamics across clients and distinguishes temporal dependence from information acquired during local learning. We establish bounds under cross-entropy loss, specializing them to Linear Next-Token Prediction (LNTP) and Self-Attention Next-Token Prediction (SANTP). These results quantify the effects of client count, communication rounds, sample size, data dimension, mixing time, and context length. In particular, they reveal a communication--generalization tradeoff: although frequent aggregation may help optimization, it can weaken generalization guarantees. They also suggest a potential advantage of distributed over centralized streaming learning under a fixed total data budget. Temporal dependence enters through the square root of the mixing time. By analyzing the history-state process, we obtain refined bounds with improved context-length dependence under suitable mixing conditions. Experiments on GPT-style language-model pretraining with TinyStories and real-world time-series forecasting illustrate how collaboration and communication frequency influence generalization under temporal dependence