Large distant gradients can mislead autoregressive forecasting updates
Large Distant Gradients Need Not Be Reliable: reliability-weighted credit assignment for long-horizon autoregressive forecasting
Machine LearningArtificial Intelligence
Summary
In long-term forecasting models that predict step-by-step, the learning updates rely on signals from far in the future. The authors found that these distant signals can become very large but also very noisy, making them unreliable for learning. They propose a method called Internal Dual-Wiener routing that weighs the learning signals based on reliability, keeping good signals strong and reducing noise. This approach improves accuracy on several forecasting tasks and challenges the idea that all long-term signals should be trusted equally.
What this means in practice
- •For time series modelers: Improve long-term forecasting accuracy by reliability-weighting gradient updates to reduce noise amplification in training.
- •For machine learning engineers: Enhance training of autoregressive sequence models by using Internal-DW to balance signal preservation and noise suppression in backpropagation.
Authors
Junhao Zhao, David Michael Simberg, Jacob Kang, Colin Connor Kurniawan, Nan Xu
Abstract
In autoregressive forecasting, long prediction rollouts provide distant supervision, but backpropagation through time (BPTT) carries gradients from those losses through many autoregressive steps. Repeated Jacobian products can make distant gradients dominate the update while amplifying predictable signal and unpredictable noise together; a large distant gradient therefore need not carry reliable learning signal. Motivated by this observation, we introduce Internal Dual-Wiener routing (Internal-DW), a principled backward-only intervention that preserves the full forward rollout and all horizon losses while reliability-weighting internal gradient routes. At each residual block, we derive bounded Wiener gains for the identity and nonlinear routes that balance preserving predictable learning signal against suppressing unpredictable variation, and estimate them from route-level gradient statistics and an explicit noise model. In a controlled system with known gradient signal-to-noise ratio (SNR), we show that distant gradients can grow even as their SNR falls, and that Internal-DW reduces held-out error in recovering predictable gradient signals and improves forecasting. On four history-dominated, weak-drive testbeds, Internal-DW reduces forecast error by 5.2%-13.8% relative to full BPTT, outperforms gradient clipping and Jacobian regularization on all four, and outperforms validation-selected truncated BPTT (TBPTT) on three. It also extends or preserves the fitted optimal training-horizon range across these four testbeds. Across the full benchmark suite, the current Internal-DW estimator has a clear applicability boundary: its benefit diminishes or reverses when usable history is limited or when the selected sampler fails to represent dominant drive-dependent variation. The results show that retaining long-horizon supervision does not require trusting every backward contribution equally.