Pooling improves forecasting while learned weighting can hurt performance
Pooling Helps, Learned Weighting Hurts In-Context: Decomposing Group Attention
Machine Learning
Summary
This paper studies a technique used to predict future data points in groups of related time series. The authors looked closely at two parts of the method: one that averages data across variables and another that learns to weight them differently. They found that simply averaging the group data generally helps predictions, but learning custom weightings can actually reduce accuracy in many cases. This insight helps understand which parts of such models work best for different prediction tasks.
What this means in practice
- •For time series analysts: Improve forecasting models by using uniform pooling instead of learned weighting to better predict data from multiple sensors.
- •For machine learning engineers: Build more robust in-context learning systems by replacing learned attention weights with uniform pooling in early network layers.
Authors
Michael Fore, James Mason Inder, Mrishika Nair, Praneetha Vaddamanu, Sharlina Keshava
Abstract
Group attention, introduced by the time series forecasting model Chronos-2, attends over the variates of a group at a fixed patch index and serves both multivariate (MV) and in-context learning (ICL) forecasting. Rather than evaluating this cross-variate attention design as a whole, we ask which part of the mechanism earns the benefit and probe its applicability to both MV and ICL regimes. By editing the attention matrix $α$ at inference we separate the two pathways a head comprises: V/O, which projects a weighted summary of the group, and Q/K, which decides the weights. Uniform pooling (V/O without any Q/K weighting) is positive on 18 of our 20 sensor-network configurations, while the learned weighting (Q/K) splits by group type: its contribution is positive or negligible for MV, but materially degrades 8 of the 10 sensor-network ICL configurations, leaving 4 of them worse than univariate inference. By isolating the impact of different layers, we find that uniforming $α$ in the first block alone improves every ICL configuration we test.