Sparse echo state networks cut model size and speed up time series tasks
Sparse Federated Echo State Networks for Time-Series Classification via Smooth-L1 Regularization
Machine Learning
Summary
Processing time-based data like sensor readings or stock prices often requires complicated computer models. The authors present a way to simplify a special type of model called an Echo State Network by making some parts sparse—that is, by zeroing out many connections. Their method keeps the model accurate while reducing its size and the computing power needed for making predictions, which is helpful for devices like smartphones or sensors with limited resources. They also show that this simpler model works well when training data is held across many devices without sharing raw data.
What this means in practice
- •For iot device developers: Enable efficient on-device processing of time-series data by deploying smaller, sparse echo state network models that reduce memory and computation needs.
- •For federated learning engineers: Train time-series models collaboratively across devices with limited data sharing using second-order federated updates that converge quickly and produce compact models.
- •For wearable technology manufacturers: Integrate lightweight and accurate time-series classifiers into wearables by using sparse echo state networks to save battery life and reduce data transmission size.$Commercial implications: This enables selling improved wearable devices with better on-device analytics and efficient data handling thanks to compressed models focused on time-series.
Authors
Gengcheng Lyu, Manuel Röder, Frank-Michael Schleif
Abstract
Echo State Networks (ESNs), as lightweight implementations of reservoir computing, provide efficient solutions for temporal data processing in resource-constrained environments. While standard ESNs offer computational efficiency, their dense readout layers inflate the stored model and the per-sample inference cost. This paper introduces a Smooth-L1 regularized ESN (SL1ESN) that promotes sparsity in the readout layer while preserving differentiability for gradient-based optimization. The method employs a twice-differentiable Smooth-L1 approximation compatible with Newton's method, which we exploit both for centralized training and for a second-order (Newton-step) federated aggregation scheme. Under a single fixed shrinkage threshold shared by every dataset and both settings, experiments on seven time-series classification benchmarks show that SL1ESN learns readouts that are $77$--$95\%$ sparse in the centralized setting and $75$--$95\%$ sparse in the federated one, while staying within $0.7$ accuracy points of the strongest dense baseline on five of the seven benchmarks in the centralized setting and within $1.6$ points on the same five in the federated one, and the second-order federated update converges in only a few rounds. The resulting sparsity compresses the stored model, the per-sample inference cost, and the server-to-client broadcast by a comparable factor. These results demonstrate the practical value of sparse readouts for edge computing and IoT deployments.