AI summaryⓘ
The authors address the problem of estimating accurate risk (covariance) matrices for portfolios when data has missing or intermittent trading information, common in small-cap stocks. They develop a neural network model that processes pairwise correlations computed on different time samples, ensuring the final covariance matrix is valid for portfolio optimization. Their model learns to adjust the correlations to produce better risk estimates and consequently more stable portfolios. Tested on 1,500 US stocks over 26 years, their method significantly reduces portfolio volatility and improves risk-adjusted returns compared to existing methods, even after accounting for trading costs. This shows their neural approach effectively handles missing data in financial risk estimation.
covariance matrixpairwise-complete estimationsmall-cap stocksMarkowitz optimizationrandom-matrix theoryneural covariance estimatoreigenvaluesrealized volatilitySharpe ratioportfolio risk
Abstract
Small-cap-inclusive equity universes contain recently listed and intermittently traded securities, so enforcing a common look-back discards a substantial fraction of the available information. Pairwise-complete estimation preserves the longest overlap for each asset pair, but the resulting correlation matrix can be indefinite because its entries are computed on different samples. This prevents direct use in Markowitz optimization and falls outside the assumptions of standard random-matrix shrinkage. We adapt a rotation-invariant neural covariance estimator to this setting. The model computes mask-aware marginal moments and a pairwise correlation matrix proxy, processes its signed spectrum, and uses a bidirectional gated recurrent unit conditioned on factor-aligned effective sample lengths derived from the overlap matrix and eigenvector loadings. It maps all eigenvalues, including negative ones, to a positive inverse spectrum. The reconstructed covariance is positive definite and is trained end-to-end to minimize five-day realized global-minimum-variance risk. We evaluate 26 expanding-window models from 2000 to 2025 on up to 1,500 U.S. equities in a closing-auction simulator with point-in-time selection, commissions, financing, corporate actions, and market impact. Across the 26-year out-of-sample period, the neural estimator reduces annualized five-day volatility by approximately 20\% and increases the Sharpe ratio by approximately 40\% relative to the next-best covariance estimator. These improvements are consistent across realized risk, risk-adjusted performance, and drawdown control, remain after the modeled execution frictions, and are supported by a 99.9\% Model Confidence Set that retains only the neural estimator.