Identifying hidden drift in noisy time series from snapshots
Identifying the Predictable Drift of a Semimartingale from Marginal Laws
Machine Learning
Summary
The paper looks at how to find the underlying trend or drift in a noisy process, even when we only see snapshots from multiple samples at different times. The authors focus on special processes that can be split into randomness and predictable components, and study how to estimate the predictable part from limited data. They tackle a challenging case where the drift depends on hidden evolving factors, and propose a mathematical way to solve this with guaranteed best solutions, while also discussing the problem’s computational difficulty. Their approach balances accuracy and complexity by breaking the problem into pieces and measuring different types of errors.
What this means in practice
- •For financial modelers: Infer hidden trends in asset price dynamics using only repeated cross-sectional market data rather than complete time series.
- •For econometric analysts: Estimate dynamic latent factors in economic indicators when only marginal distributions are observed, improving panel data inference.
Authors
Jakub Marecek, Enrico Biffis, Abigail Langbridge, Robert Shorten
Abstract
A special semimartingale admits a unique decomposition $X=X_0+M+A$ into a local martingale $M$ and a predictable finite-variation part $A$. We consider the identification of $A$ when $X$ is observed only through repeated cross-sections. The estimand is then the projection of the sampled predictable compensator onto the observable feature filtration, namely the current state together with whatever randomness is shared across the population, so that at a fixed diffusion coefficient the marginal flow identifies the drift only up to a Markovian projection. If the drift is an affine functional of an observed lag window, the joint problem is a convex quadratic programme whose solution is the pseudo-panel regression of econometrics. Our principal concern is the case, which we believe not to have been treated before, in which the drift is the output of a hidden linear dynamical system whose dynamics are themselves to be identified from the marginals. The joint problem is then a bilinear quadratically constrained programme, which we solve to certified global optimality by spatial branch and bound; with unpenalised state disturbances and a drift basis growing with the grid it is NP-hard already in latent dimension one, by reduction from $\ell^1$ rank-one matrix approximation, whereas the complexity of the deterministic system at fixed latent dimension remains open. A block-coordinate decomposition offers a cheaper alternative. For the estimator itself, we obtain rates at a fixed mesh, separated into Monte-Carlo, estimation and grid contributions.