Summary
Predicting how a system will change when you try different actions is important but hard. The authors studied models that forecast system states based on past actions and external inputs, focusing on whether these models actually reflect how changing actions affect outcomes. They found that the most accurate models often fail to predict the correct direction of change when actions shift, meaning accurate predictions don't guarantee understanding of real cause-and-effect. Introducing a new training method that explicitly rewards correct directional responses improved this aspect without hurting accuracy. This helps create models that not only predict well but also behave more like real systems.
What this means in practice
- •For clinical data teams: Improve patient state forecasts by using training losses that ensure predicted effects of treatments align with known physiological actions.
- •For infrastructure operators: Enhance system monitoring models by integrating mechanism consistency objectives to better predict responses to control actions.
Authors
Haochen Zhang, Jiaheng Guo, Zhen Xu, Zachary Plotkin, Nicholas Konz, Zhen Tan, Tianlong Chen
Abstract
A time series world model (TSWM) predicts a controlled system's state from its observed history and planned actions and exogenous inputs. Current approaches build forecasters with actions as covariates, trained and evaluated on prediction error under the executed plan. Yet world models compare unexecuted plans, but their responses to changed plans remain untested. We ask which design choices matter and whether accurate forecasters respond to changed plans as real systems do. We address both with a formalization and benchmark. The formalization separates state, actions and exogenous inputs, distinguishes continuous, mode and event actions, and introduces mechanism consistency, a metric built on declared action-state relations with known directions, such as a vasopressor raising blood pressure: it checks whether shifting an action moves the forecast in the declared direction. The benchmark consolidates eight public datasets with real actions from engineered infrastructure and clinical care, varying prediction space, plan fusion and plan encoding across seven backbones and five seeds. First, a frozen latent prediction space lowers MAE by 9.9% over observation space and gated output fusion lowers it by 12.7% over input concatenation on average, with both improving all eight datasets; temporal plan encoding changes average MAE by at most 2.2%. Second, prediction error and mechanism consistency diverge: the lowest-error configuration is at or below chance in consistency on four of five datasets with declared mechanisms, and no design choice avoids this. Finally, directional supervision, a loss penalizing the wrong-signed part of the response to a shifted action, significantly raises consistency on penalized mechanisms with no change in MAE. Together they give TSWMs a recipe: a frozen latent space and output-side fusion for accuracy, and a training objective for mechanism consistency.