Lévy Attention: Single-Pass Predictive Uncertainty for Continuous-Time Attention
2026-08-19 • Machine Learning
Machine Learning
AI summaryⓘ
The authors present Lévy Attention, a new method for deep learning models handling irregular time series data, which not only makes predictions at any time but also gives a built-in measure of how reliable those predictions are. They do this by treating the attention mechanism as a stochastic process related to a Poisson random measure, which keeps track of both the total compatibility (evidence) and the variability (disagreement) in the data. This approach provides a mathematically exact uncertainty estimate without extra computation or separate training. Their experiments show it works well, providing trustworthy uncertainty signals that improve over traditional methods with little to no loss in accuracy.
Irregular time seriesAttention mechanismPoisson random measureStochastic integralUncertainty quantificationSoftmaxCross-attentiont-PatchGNNMonte Carlo dropoutConformal prediction
Authors
Sotirios P. Chatzis, Loukas Papadoulas
Abstract
Deep models for irregularly-sampled time series answer queries at arbitrary continuous timestamps, yet report nothing about how far each answer should be trusted. We show the attention layer itself can close that gap: with the right stochastic formulation, the pass that makes each prediction also reports, in closed form and at no extra cost, how far it should be trusted. We introduce Lévy Attention, a cross-attention operator whose output is a stochastic integral against an inhomogeneous Poisson random measure: query-key compatibilities assemble an intensity over a continuous (time x channel) index space, the measure scatters atoms under it, and the output averages an interpolated value field at those atoms. In expectation it reduces to a mollified cosine-kernel attention, so it replaces a softmax layer and trains with exact gradients. What softmax discards, the Poisson construction preserves in closed form: the evidence $Λ_q$ (total compatibility mass) and the disagreement $\mathrm{tr}\,Σ_V(q)$ (value spread). An exact variance identity makes their combination $\hatσ(q)=\sqrt{\mathrm{tr}\,Σ_V(q)\,\varphi(Λ_q)}$ the root-mean-square deviation of the sampled operator, emitted by the deterministic pass with no trained head. Empirically, disagreement carries the signal, while the evidence factor swings from uninformative on dense data to strongly informative on sparse. On t-PatchGNN the operator swap costs at most 5.6% accuracy against a matched control and nothing on the sparsest dataset. The free disagreement signal improves on 20-pass MC dropout across matched five-seed suites, and $\hatσ$ scales a calibrated Gaussian whose zero-sample CRPS beats a fifty-draw sampler; a split-conformal wrapper reaches nominal coverage at every level, and one pass ranks 3,383 unseen patients by trust in 1.4 seconds.