Capability focused evaluation improves time series model selection
Beyond Static Leaderboards: Capability-Focused Evaluation with Structural Diversity for Time Series Foundation Models
Artificial Intelligence
Summary
Time series models are often tested using fixed tests that don’t fully show their true forecasting ability on different kinds of data patterns. The authors developed CaFE, a new way to evaluate these models by creating diverse test samples based on different temporal features from the original data. This method highlights each model’s strengths and weaknesses on specific time series structures instead of giving just one overall score. Their tests show that choosing models based on this detailed evaluation leads to better forecasting performance on real-world data.
What this means in practice
- •For data science teams: Select time series forecasting models tailored to specific temporal patterns to improve prediction accuracy in real-world applications.
- •For financial analysts: Assess forecasting models based on diverse temporal features found in financial time series to better anticipate market trends.
Authors
Hongyin Zhang, Mingyi Xiang, Rongzhao Chen, Yankai Wang, Jialin Qiao, Xiangdong Huang, Xiaoou Ding, Chen Wang, Zhongyi Pei, Jianmin Wang
Abstract
Static public benchmarks for time series foundation models (TSFMs) typically have fixed and narrow test distributions. Their repeated reuse encourages benchmark-specific tuning, making reported performance reflect adaptation to the fixed test set rather than genuine forecasting generalization. In addition, a single aggregate score cannot fully characterize model capabilities across distinct temporal structures and input relationships. We propose CaFE (Capability-Focused Evaluation), a refreshable evaluation extension grounded in original benchmark data that enriches static benchmarks with structural diversity and enriches single-score black-box evaluation with multi-dimensional, capability-oriented assessment. CaFE first profiles the original benchmark and defines eight Structural Features (SFs) that characterize its temporal structure. For each SF, it resamples the original data according to its structural support, removes edge-of-distribution samples, and augments the retained samples by varying the SF's salience, producing samples with different structural profiles. After filtering out out-of-distribution samples while preserving diversity, CaFE yields a refreshed evaluation set. TSFMs are then evaluated on this set using both conventional forecast-error metrics and intervention-response metrics, giving each model a per-feature capability profile alongside endpoint accuracy. Experiments on GIFT-Eval and FEV-Mini20 show that CaFE yields stable evaluation results and reveals per-feature strengths and weaknesses that a single aggregate score hides. Moreover, on genuine data filtered by structural condition, models selected by CaFE outperform those chosen by the aggregate leaderboard. CaFE therefore makes model selection conditional on the structural response of interest.