Predictor geometry limits learning and decision quality improvement
Jacobian Rank Collapse in Decision-Focused Learning
Machine Learning
Summary
Decision-focused learning aims to train models by directly improving the decisions they make rather than just predicting accurately. The authors found that the way model parameter updates happen is often restricted by the shape (rank) of the gradient information, which can limit learning effectiveness. They showed that even when the model seems to have many parameters, the actual directions available for improvement can be very few, sometimes just one. Their experiments across tasks like finance and optimization problems suggest that these geometric limits restrict how much decision quality improves compared to simpler prediction methods.
What this means in practice
- •For financial modelers: Design forecasting methods that consider limited update directions to improve decision outcomes in investment strategies.
- •For operations research teams: Understand constraint effects on learning gradients when training models for combinatorial problems like shortest path or knapsack optimization.
Authors
Aojie Yuan, Haiyue Zhang, Zijian Su
Abstract
Decision-focused learning (DFL) trains predictors through downstream objectives, but a different loss need not provide an independent parameter-update direction. We characterize this restriction through the predictor Jacobian, using sparse index tracking to distinguish the covariance entries read by the optimizer from the parameter directions available to learning. Rank-one Jacobians make nonzero per-example gradients collinear; a conditional spectral bound describes near-collinearity. A batch-subspace characterization and counterexamples show why these local statements imply neither common minimizers nor collinear batch updates. Experiments examine when geometry translates into decision quality. Across 38 one-parameter equity configurations, DFL gains over MSE remain below 1.8%; a 385-parameter conditional predictor also has pointwise rank one. In validation-tuned shortest-path and knapsack experiments, full-capacity SPO+ reduces mean regret by 11.6% and 10.6%, respectively; only knapsack survives correction across eight comparisons. The capacity contrast persists on fresh datasets across batch orders and training budgets. Holding expressivity fixed, invertible coordinate scaling lowers spectral effective rank and ordinary SGD gains; compensating for the scaling restores the original trajectories. Financial forward-target controls separate forecast accuracy from decision quality; a matched neural comparison finds no aggregate DFL advantage in the tested architecture. These findings distinguish local rank restrictions, coordinate-dependent optimization and predictive accuracy. Predictor geometry helps explain available learning directions, while held-out decision quality remains the test of practical benefit.