Reservoir computing explained through classical observer theory insights
A KKL Observer Perspective on Reservoir Computing
Machine Learning
Summary
Reservoir computing is a machine learning method that helps predict and control complex systems, but why it works so well and how to build good versions of it have been mysteries. This paper shows that reservoir computing is actually an extension of an older, well-studied mathematical problem called the KKL observer design from control theory. By connecting these ideas, the authors explain why reservoir computing can forecast data accurately and why simple techniques to train it often work. This link to classical theory provides new ways to understand and improve reservoir computing.
What this means in practice
- •For control system engineers: Design reservoir-based models with guaranteed stability and uniqueness for improved dynamic system forecasting tools.
- •For physical computing developers: Create more reliable reservoir computing architectures by applying classical observer theory to hardware implementations.
A theory result. No direct application yet.
Authors
Anastasia Bizyaeva, Fernando Castaños, Jaime A. Moreno
Abstract
Reservoir computing (RC) is a machine learning technique for data-driven modeling of dynamics for forecasting and control, primarily studied in computer science and physics literature with promising applications in neural network learning, physical computing, and neuroscience. Why reservoirs learn and how to choose good reservoir architectures are considered important open questions. We show that the RC problem is mathematically an extension of a classic problem in systems and control theory, the Kazantzis-Kravaris-Luenberger (KKL) observer design problem. As a consequence, many of the questions considered open for RC stand to benefit from a large body of theory in the mature KKL literature, non-exhaustively including on questions of embedding, transverse stability, local and global uniqueness guarantees, and effective data-driven solution constructions. Elaborating on this connection, we show that the surprising forecasting ability of reservoirs is in fact a direct consequence of the well-known observer internal model principle, derive an upper bound on the prediction error over a fixed forecast horizon, and provide a partial explanation for why linear readout training in RC works reasonably well. This work illustrates how classical ideas from systems and control can provide strong theoretical backing and open new questions for modern machine learning methods.