Benchmarking system 1 models shows mixed results on time-series tasks
Benchmarking System One Models for Time-Series Decision Making: A Case Study of Jev
Artificial IntelligenceMachine Learning
Summary
Time-series data involves information collected over time, like stock prices or temperature readings, which require quick and accurate decisions. The authors study how decision-focused models called Structured System 1 models perform on various time-series tasks using a tool named Jev. They find these models do well on some tasks with clear time patterns but struggle with predicting stock movements and many numerical sequences. Their work suggests areas to improve these models for better time understanding and decision-making.
What this means in practice
- •For financial analysts: Use evaluation insights on system 1 models to improve automated stock movement prediction tools by addressing model weaknesses in time-series data handling.
- •For industrial monitoring teams: Implement decision-focused time-series models to predict events like oil temperature changes, leveraging tested model capabilities on explicit temporal patterns.
Authors
Jianye Wang, Kexin Zhang, Jiayin Xiong, Hao Xue
Abstract
Time-series tasks require both accurate predictions and fast decisions. Advances in large models have introduced decision-oriented models that directly output answers and their probabilities, which we call Structured System 1 models. Their applicability to time-series tasks remains underexplored. Using Jev 1.13.0 as a case study, we evaluate time-series classification, event type prediction, oil temperature direction and stock movement prediction, converting continuous forecasting targets into discrete decisions to fit the model interface. We establish a unified evaluation framework using accuracy and AUC. Results show stronger performance on selected explicit temporal patterns and event type prediction, but near-chance discrimination on many numerical sequence classification tasks and stock movement prediction. Some datasets also reveal a gap between ranking quality and final decision accuracy. These findings motivate improvements in temporal representation, domain adaptation, and decision-score alignment. Standardized sample construction, model calls, and result recording support extending the evaluation to future models.