Causal optimal-transport attention enables longer context length extrapolation
Causal Optimal-Transport Attention: Positional Priors and Length Extrapolation
Machine Learning
Summary
Long-range attention in AI language models often struggles when trying to understand longer texts than those seen during training. This work studies a specific type of attention called causal optimal-transport attention, which respects the natural order of words but has challenges balancing attention across long sequences. The authors analyze how this method spreads focus over positions and show ways to control the attention so that models better handle longer texts without losing important context. They test their approach on language models and find it improves predictions on longer inputs compared to standard methods.
What this means in practice
- •For nlp engineers: Deploy language models capable of more accurate predictions on longer texts than those seen during training by using causal optimal-transport attention variants.
- •For speech recognition developers: Improve the handling of long audio contexts in transformer-based models by incorporating positional priors from optimal-transport attention mechanisms.
Authors
Ming Lian
Abstract
Causal masking obstructs the column balancing used by optimal-transport (OT) attention. We study a prefix-consistent unbalanced operator and characterize the positional prior it induces. Under bounded scores, for any fixed balancing strength $ρ<1$, its fixed point dilutes attention: every fixed window receives a vanishing share as the context grows. For constant scores, we derive an exact scale-invariant prior. We then classify positional encodings by whether their distant attention mass remains bounded away from one. Discrete window-mass bounds quantify the finite-length dilution of summable log-distance priors near exponent one. Preregistered experiments with 25M- and 100M-parameter language models establish improved loss extrapolation over softmax; exploratory comparisons favor an explicit distance bias at a fraction of the cost. CABLE outperforms RoPE with evaluation-time YaRN on every seed at three training lengths. Changing only the bias beyond the training length moves long-context loss in the predicted direction on all 30 tested checkpoints. Even with exponents constrained above one, many heads approach the summability boundary during training. A margin selected by prior-only calculations for a specified window, length range and growth tolerance controls distant-mass growth in the tested setting. These results connect loss extrapolation to positional decay over the lengths actually evaluated.