Models improve forecasts when text agrees with sensor data trends

Aligned, Conflicting, or Unspecified? A Diagnostic Benchmark for Multimodal Time Series Forecasting

Artificial Intelligence

Summary

Forecasting systems often combine sensor data with related text like weather reports or news, but it is unclear how well they handle situations when the text agrees or disagrees with the sensor data. The authors created a benchmark with examples labeled as aligned, conflicting, or unspecified based on how the text relates to the sensor history. They found that when text aligns with the data, models forecast better, but conflicting text can harm accuracy unless carefully combined. They also designed a simple rule to limit excessive forecast changes caused by conflicting text, which improved accuracy for some models.

What this means in practice

  • For weather forecasting teams: Improve temperature forecasts by integrating aligned textual weather descriptions with sensor data while controlling revisions from conflicting reports.
  • For financial analysts: Use the benchmark to evaluate how text-based news influences market trend predictions alongside numeric market data, identifying beneficial combinations.

Authors

Xintong Wang, Haihan Nan, Ruidong Li

Abstract

Artificial Intelligence of Things (AIoT) systems increasingly combine sensed time series with external text such as expert reports and news, yet how reliably models use text that agrees or conflicts with the sensed history remains unclear. We introduce the first real-world diagnostic benchmark organized by text-history relationship: 1,762 text-window pairs across sensor telemetry (Weather), web-service traffic (Wikipedia), and market data streams (Finance), each labeled Aligned, Conflicting, or Unspecified by whether the text's direction agrees with the historical trend, opposes it, or is not stated. Across ten pretrained models evaluated zero-shot, we investigate text selection and combination under text-history relationships (RQ1) and error diagnosis and horizon-scaled revision control (RQ2). For RQ1, we compare two types of text on paired Weather windows: one directly forecasts temperature changes, while the other describes related weather conditions, such as winds and fronts. Under Aligned, the first type yields the lowest error for most models. Under Conflicting, it increases most models' error above the no-text baseline; we therefore add only its stated magnitude and timing to the second type. Compared with either type alone, this combination lowers mean absolute scaled error (MASE) for six models and raises directional accuracy (DA) for seven. For RQ2, an error decomposition attributes the error increase to oversized revisions, and a training-free rule that limits them outperforms forecasts both with and without text for the four models with this trade-off, lowering MASE by 4.49%-21.98% and raising DA by 2.99%-9.71%. Our dataset is available at https://github.com/Shelton1202/ACU-Bench.