Multivariate forecasting improved by fixing conflicting training signals
Not All Variables Agree: Reliability-Aware Variable-Wise Gradient Surgery for Multivariate Time-Series Forecasting
Machine Learning
Summary
Predicting multiple things over time often uses a training method that mixes all errors together, hiding how different parts affect learning. The authors found that gradients, which guide training, often disagree between variables, but this disagreement alone does not explain which parts perform worse. They created a method called Per-Variable Surgery that looks at variable-specific signals during training and adjusts the learning directions accordingly. This approach improves forecasting accuracy by keeping the training focused on variable-level conflicts and agreements that are otherwise lost.
multivariate time-series forecastinggradientoptimizerbackpropagationmean squared errormean absolute errorcosine similaritytraining conflictvariable-wise gradientloss function
Authors
Jinwoo Park, Hyeongwon Kang, Pilsung Kang
Abstract
In data-driven training, multivariate time-series forecasting is usually optimized with a scalar loss averaged over samples, variables, and horizons. This averaging is convenient, but the optimizer sees only the aggregated gradient, which does not reveal whether the variable-wise contributions align or oppose one another. To quantify how often this disagreement arises, we measure the variable-wise gradients directly and find that 30.6% of their pairwise cosine similarities are negative on average across seven datasets. However, conflict and harm are not the same thing. Under shared training 35 of the 64 variables do worse than a full-input single-target oracle, and the harmed fraction is not reliably predicted by how often gradients conflict. We propose Per-Variable Surgery (PV-Surgery), an optimizer-side training strategy for backbones with cache-compatible layers. One backward pass builds variable-wise gradient proxies from output-side signals and keeps the pointwise forecasting loss. Reliability-aware selection targets layers whose proxy sums closely approximate their shared-gradient slices. Conditional pooling forms anchor and conflict pools without dropping variables. Common-direction surgery aligns variable or pooled gradients with their normalized mean and restores input norms to avoid reweighting. In experiments across five backbones, seven datasets, and four horizons, PV-Surgery lowers MSE by 3.61% and MAE by 2.93% on average. For multivariate forecasting, this indicates that the variable-wise structure hidden by mean-loss training is a usable optimization signal.