Time series models improve by routing on raw inputs before normalization
Raw-Routed Mixture of Adapters: A Causal Intervention for Routing Collapse in Time Series Foundation Models
Machine Learning
Summary
Time series models often struggle to adapt to new patterns because they use a single adjustable part on top of a fixed main model. A common solution is to add multiple specialized parts and decide which to use for each input, but this fails when the model normalizes data beforehand, causing the decision process to collapse to always choosing the same part. The authors identify that the normalization step removes important differences needed for routing. They propose a fix that routes decisions based on the raw input before normalization, which consistently improves model performance across various tests.
What this means in practice
- •For time series engineers: Improve model adaptation to diverse time series regimes by adding routing on raw inputs before normalization.
- •For financial modeling teams: Enhance forecasting accuracy by integrating raw-input routing mixtures in frozen time series models.
Authors
Hung Phan, Thuy T. Nguyen, Minh Ngoc Dinh, Nhat-Quang Tran
Abstract
Time series foundation models (TSFMs) commonly adapt to new data by attaching a single trainable head to a frozen backbone, a one-size-fits-all setup that underfits heterogeneous regimes. Replacing the head with a mixture of experts is the standard upgrade, but on instance-normalized backbones (the dominant TSFM design class) it fails: routing entropy collapses to zero and one expert absorbs every input, a failure we call normalization-induced routing collapse. Standard MoE rescue mechanisms do not repair it, because the cause is in the router's input, not its optimization. Pre-encoder normalization strips the statistics a router would need to tell regimes apart. A mutual-information decomposition makes this precise and yields a signal-ratio that, computed before training, predicts dataset vulnerability (Spearman $ρ= -0.88$). Eight causal controls, including a vision-modality replication, isolate instance normalization as the cause. The prescription is a minimal causal intervention: Raw-Routed Mixture of Adapters (RR-MoA), which routes on the raw, pre-normalization input. Under a strictly frozen backbone, RR-MoA wins 54/54 comparisons against the strongest fixed adapter and significantly outperforms LoRA, TRACE, AdaMix, and full fine-tuning. The effect generalizes across six backbones and an imputation task. Frozen RR-MoA also beats full fine-tuning by 12-79% (the Frozen Paradox); two architecturally distinct variants confirm the principle generalizes beyond this specific router.