Lightweight gate improves time-series forecasting with external data
A Lightweight Plug-in Gate for Transformer-Based Time-Series Forecasters
Machine Learning
Summary
Time-series forecasting predicts future data points based on past values and outside information, like weather or traffic. The paper looks at a simple way to control how extra information is fed into a popular forecasting model called a Transformer. The authors add a small gating mechanism that decides which parts of the outside data to use before the model processes it. Testing on multiple datasets shows this gate works about as well as the original models but can also limit unnecessary data use without hurting accuracy much.
What this means in practice
- •For energy grid operators: Improve minute-level electricity demand forecasts by regulating external data entry in Transformer models to maintain accuracy while reducing unnecessary input processing.
- •For transportation planners: Enhance traffic flow predictions by selectively admitting external variables through a lightweight gating module in Transformer-based forecasting systems.
Authors
Hongkai Zhuang, Tao Huang, Chen Hou
Abstract
Covariate-rich time-series forecasting requires deciding how external variables enter the target forecasting path. Existing Transformer-based forecasters usually build a covariate representation and pass it to the encoder without an explicit admission stage. This paper studies pre-encoder covariate admission as an input-side interface that regulates that representation immediately before encoder processing. We implement the interface with a lightweight representation-level pre-encoder gate that assigns sigmoid scores to representation units, and we also study a usage-regularized variant that penalizes average admission. The interface is evaluated as a plug-in module for TimeXer, Inverted Transformer (iTransformer), and Patch Time Series Transformer (PatchTST) under a zero-extra-tuning protocol, where each gated model inherits the corresponding baseline configuration. Experiments on the Electricity Transformer Temperature minute-level (ETTm1 and ETTm2) datasets, Traffic, Energy, and influenza-like illness (ILI) include paired forecasting comparisons, gate-placement ablation, initialization ablation, controlled covariate-admission analysis, and a variance inflation factor (VIF)-informed permutation feature importance (PFI) diagnostic case study. In the tested settings, the gate is competitive with the corresponding baselines, and the usage penalty reduces average admission scores while keeping forecasting errors close to the unpenalized TimeXer setting.