Slower-growing attention kernels improve learning from heavy-tailed data
Attention Kernels for Learning Maps Between Heavy-Tailed Measures
Machine Learning
Summary
Standard transformer models use a softmax function that can overemphasize large inputs, which causes problems when dealing with data that has heavy tails—rare but extreme values that appear more often than normal. The authors found that replacing the softmax with slower-growing functions helps avoid these issues when learning from such data. They tested these new functions on different tasks, showing they prevent the models from collapsing and work well, especially on difficult cases. This suggests a better way to design transformer-based models that learn from data with extreme variations.
What this means in practice
- •For machine learning engineers: Design transformer models that reliably learn mappings from datasets with heavy-tailed statistical properties by using slower-growing attention functions.
- •For data scientists in finance: Improve the stability of models analyzing financial data with heavy-tailed distributions by adopting modified attention kernels to better capture extreme events.
Authors
Kailen Hargenrader, Edoardo Calvello, Bohan Chen
Abstract
Operator learning on probability measures can be accomplished with transformers. For measures with polynomial tails, the exponential weighting in softmax can make the corresponding measure-level attention integrals diverge. This motivates replacing the exponential with slower-growing functions. We construct two benchmarks for operator learning on measures with closed-form targets. We use these benchmarks to study attention kernel growth and data transformation in post-norm transformers. Without data transformation, the softmax models exhibit ensemble collapse on both heavy-tailed benchmarks, while the three slower-growing kernels avoid collapse. Symlog preprocessing allows softmax to avoid collapse on the matrix inverse task but not on the sheared swap task. On the Gaussian control, all four kernels perform similarly. We also examine how sample size affects the sensitivity of empirical energy and Wasserstein distances to tail differences. These results support slower-growing attention kernels as an effective design choice for post-norm transformers learning from heavy-tailed ensembles.