Forecast accuracy may not improve spare parts order fulfillment
Accuracy Is Not Service: A Decision-Aware Benchmark for Intermittent-Demand Forecasting
Machine Learning
Summary
Companies that supply spare parts get paid only when entire customer orders are fulfilled, but many forecasting methods focus on predicting individual parts accurately. The authors found that the best methods for forecasting each part do not always lead to the best overall order fulfillment, especially when parts demand is irregular and sparse. They showed that the direction of forecast bias, like predicting too many parts during no-demand times, affects overall service more than precise accuracy. The authors also provided a simple correction to improve order fill rates without retraining the forecasting model.
What this means in practice
- •For supply chain planners: Improve spare parts inventory forecasts by focusing on bias direction to increase order fulfillment rates without retraining models.
- •For logistics managers: Assess forecasting methods based on order-level service performance rather than line-item accuracy to better align contracts and payments.
Authors
Joo Ern Chin, Shih-Fen Cheng, Aldy Gunawan
Abstract
A contract-logistics spare-parts operator is paid on order-level service: an order counts only if every requested line is fulfilled, yet forecasters are selected based on line-level forecast accuracy. This disconnect matters when demand is intermittent and lumpy, histories are short, and lead times span months. We benchmarked 38 forecasting methods spanning classical, intermittent-demand, machine-learning, deep-learning, and pretrained foundation models. A common decision-aware protocol evaluates them on an industrial panel drawn from a live contract and two public datasets. Forecast-accuracy rank and order-service rank are negatively correlated on the industrial panel, at -0.555, across methods evaluated on 20,330 real multi-item orders. Service is more closely associated with the direction of cumulative forecast bias, including over-prediction during zero-demand periods, than with point accuracy. Examining bias in Chronos-2's instance normalization yields a training-free correction that lifts the per-material fill proxy from 77.5% to 92.0% (14.5 percentage points) at the 90% policy target and raises the complete-order fill rate from 54% to 63%. For reproducibility, we release RUF (Regenerate-Until-Fidelity), a method for generating fidelity-certified synthetic panels on which the findings reproduce. For intermittent demand, the lowest-error forecast need not deliver the highest service. Bias direction helps explain this gap, which can be reduced without retraining.