Nystrom attention matches full attention in stock prediction models
Nyström Attention Matches Full Attention for Cross-Sectional Stock Prediction
Machine Learning
Summary
Predicting stock movements involves understanding relationships between many stocks at once. The authors studied an important part of a stock prediction model that looks at these relationships, called attention, and found it behaves almost uniformly but with subtle patterns. They discovered a way to approximate this complex attention efficiently using a low-rank method called Nystrom attention without losing accuracy. This suggests that the key value in understanding stock relationships lies in a compressed, global way of sharing information rather than focusing on sparse or simple connections.
What this means in practice
- •For quantitative finance teams: Implement efficient cross-stock relationship models using Nystrom attention to reduce computational cost without losing predictive accuracy in stock prediction systems.
- •For financial data engineers: Optimize large-scale stock prediction pipelines by replacing costly full attention with low-rank approximations validated for equivalence on up to 800 stocks.
Authors
Kunhan Guo
Abstract
MASTER's inter-stock multi-head attention -- the module responsible for modeling cross-sectional stock relationships -- accounts for 42.5% of model parameters and 25% of predictive value. We systematically decompose this module and uncover a surprising structure: the learned attention is near-uniform (perplexity 278/300), yet forcing exact uniformity eliminates all cross-sectional discrimination. Spectral analysis resolves this paradox: the deviation from uniformity is low-rank (effective rank ~65, top-10 modes capture 96.5% of energy), explaining why sparse approximations consistently fail while Nystrom low-rank attention (m=32 landmarks) matches full O(N^2) attention at O(mN) cost -- certified equivalent via TOST at both N=300 (5 seeds, Rank IC p=0.003) and N=800 (10 seeds, Rank IC p=0.034). Additional findings include: (i) attention anti-correlates with return similarity (Spearman rho = -0.614; on the industry-labeled subset, -0.645 unconditionally and -0.627 after controlling for industry, beta, and volatility), suggesting complementarity-seeking rather than correlation mining; (ii) all graph-based alternatives degrade performance, with hard masking worse than complete module removal; and (iii) at N ~ 3,500 with adapted architectures, no cross-stock module (GCN, Nystrom, or MASTER-style pipeline) significantly outperforms a per-stock LSTM baseline (n=4 seeds), indicating that the benefits observed at smaller scales do not trivially transfer. These results establish that the inter-stock attention's value resides in a compressible, dynamic, near-global redistribution that rewards low-rank approximation but resists sparsification.