UCR time series archive tasks often differ from real classification goals
Fifty-Two Shades of Pig: What the UCR Archive Really Asks Classifiers to Do
Machine Learning
Summary
The UCR Time Series Classification Archive is a popular collection of data used to test time-based pattern recognition methods. The authors found that many datasets in the archive ask questions different from their original or intended purpose, such as identifying individual pigs instead of detecting medical conditions. They highlight problems like unrealistic experimental setups and flawed test data splits in some datasets. Their findings suggest that overall benchmark scores may not always reflect how useful these methods are for real-world tasks, calling for more careful evaluations.
What this means in practice
- •For machine learning engineers: Adjust evaluation strategies by verifying real-world relevance of benchmark datasets when developing time series classifiers.
- •For medical device developers: Use caution interpreting classification results from existing benchmarks when designing systems for early detection of patient conditions.
A position paper. It proposes an approach and reports no results.
Authors
Omar Swelam, Jaris Küken, Shi Bin Hoo, Frank Hutter, Lennart Purucker
Abstract
The pigs were bleeding. That was the point of the experiment: could airway pressure reveal hemorrhage before it became critical? In the UCR Time Series Classification Archive, the de facto benchmark for Time Series Classification (TSC), the same recordings ask a different question: "which of 52 pigs produced this trace?". The dataset PigAirwayPressure is not an outlier. Although the archive is the standard benchmark, whether its tasks resemble the problems TSC is meant to solve has rarely been examined. We find that several datasets are repurposed from their original task, built from toy setups, or have flawed test splits. We further question how representative the archive's ordered non-temporal sequences (e.g. spectra, shape outlines) are of real TSC, and the potential downsides from the decomposition of multivariate problems. Our findings call for more careful interpretation of archive-wide scores and greater attention to deployment-relevant evaluation. The audit is at https://github.com/automl/ucr-provenance-audit.