Summary
Understanding how well a model can predict future events when only partly observing a system is a tricky problem. This paper by the authors shows how to calculate meaningful error limits for such predictions in systems where things evolve over time in a linear way and have noise that behaves nicely (called sub-Gaussian). These limits help us know how accurate the model's predictions are, even with limited data, and can also tell us how close we are to the true system parameters. Additionally, since these linear systems are related to some types of neural networks, this work might help improve understanding of those models too.
PAC-Bayesian boundslinear time-invariant systemsstate-space modelpartially observed systemsstochastic dynamical systemssub-Gaussian noiseprediction errorparameter estimation errorfinite-sample analysisrecurrent neural networks
Authors
Mihaly Petreczky, Mohamad Al Ahdab, John Leth
Abstract
In this paper we derive a Probably Approximately Correct (PAC)-Bayesian error bound for partially observed linear time-invariant (LTI) stochastic dynamical systems in state-space form with inputs and sub-Gaussian noise. Such bounds are widespread in machine learning, and they are useful for characterizing the predictive power of models learned from finitely many data points. The bound derived in this paper relates the expectation of prediction errors with the prediction error generated by the model on the data used for learning. In addition, we show that it can also be used to derive bounds for the parameter estimation error. In turn, this allows us to provide finite-sample error bounds for the prediction error and parameter estimation error for a wide class of system identification algorithms. Furthermore, as LTI systems are a sub-class of recurrent neural networks (RNNs), these error bounds could be a first step towards PAC-Bayesian bounds for RNNs.