PAC Bayesian error bounds improve learning of noisy linear dynamical systems
PAC-Bayesian Bounds for Learning Partially Observed Stochastic Linear Time-Invariant State-Space Systems with Inputs and Sub-Gaussian Noise
Machine Learning
Summary
Predicting how systems change over time can be hard when you can only see part of the system and measurements have noise. The authors derived a way to estimate how accurate predictions will be after learning from limited noisy data. Their approach links the average error on new data to the error on the data used for learning, helping to understand and limit mistakes. This also helps measure how well the underlying system parameters are learned. Since linear time-invariant systems are a simple kind of recurrent neural network, this work could help develop error bounds for more complex models too.
What this means in practice
- •For control engineers: Estimate prediction and parameter errors when identifying real-world noisy linear systems from limited data.
- •For machine learning practitioners: Use error bounds to evaluate learning algorithms for recurrent neural networks inspired by linear dynamical systems.
Authors
Mihaly Petreczky, Mohamad Al Ahdab, John Leth
Abstract
In this paper we derive a Probably Approximately Correct (PAC)-Bayesian error bound for partially observed linear time-invariant (LTI) stochastic dynamical systems in state-space form with inputs and sub-Gaussian noise. Such bounds are widespread in machine learning, and they are useful for characterizing the predictive power of models learned from finitely many data points. The bound derived in this paper relates the expectation of prediction errors with the prediction error generated by the model on the data used for learning. In addition, we show that it can also be used to derive bounds for the parameter estimation error. In turn, this allows us to provide finite-sample error bounds for the prediction error and parameter estimation error for a wide class of system identification algorithms. Furthermore, as LTI systems are a sub-class of recurrent neural networks (RNNs), these error bounds could be a first step towards PAC-Bayesian bounds for RNNs.