Quantile forecasts that work well for many loss measures online
Online Quantile Omniprediction for Proper Losses
Machine Learning
Summary
People often need prediction methods that perform well under different ways of judging accuracy. The authors develop algorithms that create forecasts as sets of quantiles, which are values marking probabilities, and ensure these forecasts remain accurate across many proper loss measures. These algorithms work in an online fashion, improving as new data arrive, and can adapt to changing conditions. One example they show is using these methods to combine COVID-19 hospitalization forecasts from different experts.
What this means in practice
- •For hospital data teams: Combine multiple expert hospitalization forecasts efficiently to provide reliable multi-criteria risk assessments.
- •For financial risk analysts: Create predictions useful under many different risk measures simultaneously to inform investment decisions.
Tested on one dataset.
Authors
Hyunsuk Kim, Isaac Gibbs, Michael W. Mahoney, Ryan J. Tibshirani
Abstract
The task of constructing a predictor that is simultaneously accurate under multiple evaluation metrics has been popularized recently under the name omniprediction. Prior work in this area has largely focused on binary prediction tasks, and existing results for multiclass and continuous outcomes typically suffer from slow error rates. In this paper, we consider forecasts expressed as a set of quantiles over a discrete but otherwise arbitrary set of probability levels. We develop sample-efficient algorithms for constructing forecasts that are accurate simultaneously across all proper losses that satisfy mild regularity conditions, where proper losses are those minimized by predicting the true quantiles. Our algorithms are online and can operate in a dynamic environment, while simultaneously taking multiple losses into account, and their forecasts offer the potential to provide diverse end-users with rich information for evaluating risk and informing a range of practical decision-making tasks. As an example, we demonstrate how our procedure can be used to ensemble quantile forecasts of COVID-19 hospitalizations made by experts during the pandemic.