Estimating future states in complex time sequences with low-dimensional models
Generative sequence modeling for infinite memory processes via predictive states
Machine Learning
Summary
Predicting what will happen next in complex sequences like weather or stock prices is hard because many methods assume simple patterns or short memories. This paper introduces a new way to make predictions even when the sequence depends on a long and complicated history. The authors show that if the past information can be summarized efficiently, their method can learn quickly and accurately. They use deep learning to build and test their approach, providing both theoretical backing and experimental proof.
What this means in practice
- •For data engineers: Build better predictive models for time series with complex long-range dependencies without needing strict assumptions on memory length.
- •For financial modelers: Improve forecasts of market behavior by efficiently capturing long-term nonlinear interactions in multivariate financial data.
Authors
Michael Wieck-Sosa, Cosma Rohilla Shalizi
Abstract
We consider estimating the one-step-ahead conditional distribution of a multivariate stochastic process. Many existing approaches rely on assumptions such as finite-range memory, sparsity, or additivity, which can be poorly suited to processes with long-range nonlinear interactions. However, without such structural assumptions, nonparametric estimation is challenging due to the curse of dimensionality. To address this challenge, we introduce a new estimation approach based on the predictive states of a process, possibly with infinite-range memory. We show that our estimator achieves fast convergence rates when the past history can be compressed into a low-dimensional statistic that is sufficient for predicting the future. Specifically, we show that the statistical complexity of the estimation problem is determined by the intrinsic dimension of the predictive state space. We establish guarantees for an instantiation of our method based on deep neural network estimators, and we support these theoretical results with experiments.