Bayesian modular models improve time series forecasting and detection
Modular Bayesian Foundation Models: Bayesian time series encoders for OOD detection, routing and forecasting
Machine Learning
Summary
Industrial machines produce many kinds of changing data over time, making it hard for one model to understand all conditions well. The authors designed a system that uses many small specialized Bayesian models to better recognize and predict these changes efficiently. These models can decide which specialist to use for new data by estimating their own uncertainty, which helps in detecting when the data is unusual. Their approach uses much less storage and achieves accurate predictions with good uncertainty estimates compared to existing methods.
What this means in practice
- •For industrial data engineers: Select specialized models dynamically to predict and detect changes in machine data streams efficiently with less storage use.
- •For energy grid operators: Improve forecasting accuracy and detect unusual conditions in power usage time series by routing data through Bayesian experts.
Authors
Mame Diarra Toure, Quang Hung Pham, Mohamed Ryad Zemouri, David Stephens, Luc Vouligny, Martin Gagnon
Abstract
Industrial systems continuously generate time series that vary in sampling frequency, scale, dynamics, and operating regime. Consequently a single time-series foundation model may not be able to represent all regimes effectively and retraining a large model for every new dataset or regime shift is computationally expensive. Modular foundation models address this issue by reusing specialised encoders trained on diverse datasets and by allowing the expansion of capabilities through the addition of new experts. This approach requires compact experts to minimize storage cost and a proper selection mechanism of experts for unseen data. We propose a modular foundation model framework that combines end-to-end Bayesian encoder training with uncertainty based routing. Using Singular Bayesian Neural Networks (SBNNs), we build a collection of compact Bayesian encoders whose epistemic uncertainty guides label-free selection. At the selected ranks, our Bayesian collection uses 63.8\% less storage than its deterministic counterpart. We evaluate this framework on time-series data through distribution shift, encoder selection, and probabilistic forecasting. Across seven incoming datasets, uncertainty-based routing is competitive with distance-based routing in terms of forecast MSE, while reducing average ECE from 0.086 to 0.072 and average NLL from 0.907 to 0.836. It also increases PICP90 from 82.28\% to 89.03\%. In a separate model-family comparison, we evaluate the SBNN pipeline against full-rank Bayes by Backprop, deep ensembles, and MC Dropout. The SBNN pipeline achieves an average MSE of 0.390, comparable with 0.396 for deep ensembles, 0.389 for MC dropout, and 0.443 for the full-rank Bayesian model. It uses 63.9\% fewer parameters than MC dropout, 81.9\% fewer than the full-rank model, and 92.8\% fewer than the deep ensemble.