LUCID improves causal links in time series despite hidden influences
LUCID: Learning Under Confounding for Inference and Discovery in Time Series
Machine Learning
Summary
Hidden common causes often confuse methods that try to find true cause-and-effect links in time series data, making it look like there are connections where none exist. The authors created LUCID, a technique that first figures out how hidden causes affect the data and then removes their influence to reveal the real causal relationships. LUCID can be used with different discovery methods and consistently improves their accuracy. It works well across many challenging situations, including sudden changes and unusual data behaviors.
What this means in practice
- •For financial analytics teams: Generate more accurate causal graphs from financial time series despite hidden market factors affecting multiple assets.
- •For industrial process engineers: Improve fault detection by identifying true causal interactions in sensor data clouds subject to hidden system influences.
Authors
Mohammad Fesanghary
Abstract
Unobserved common causes are pervasive in real-world time series and can induce spurious associations that causal discovery methods mistake for direct edges. We propose LUCID (Learning Under Confounding for Inference and Discovery, a regime-adaptive deconfounding layer that first estimates the confounding regime from data using a Marčenko--Pastur spectral router, then applies a deconfounding strategy matched to that regime. When the spectrum indicates pervasive factor confounding, LUCID attenuates factor-dominated variation and recovers contemporaneous (lag-$0$) structure from the resulting innovations, with edge selection calibrated against a data-driven edge-free null. Rather than being tied to a particular discovery algorithm, it can wrap existing discovery engines; we demonstrate consistent improvements across three such methods. On a diverse synthetic out-of-distribution benchmark spanning changes in confounder strength and sparsity, loading density, lag structure, volatility dynamics, edge heterogeneity, persistence, intermittency, and tail behavior, LUCID achieves the best family-weighted directed, lag-resolved graph $F_1$ ($0.60$), improving over the strongest baseline by $0.19$ absolute ($\approx\!46\%$ relative). Its advantage widens relative to looser lag-collapsed scoring, and remains robust under intermittent and heavy-tailed confounding. Code reproducing the method, the benchmark generators, and every reported experiment is available at https://github.com/bloomberg/causal-ts.