World models improve prediction with timing of irregular sensor updates
World Models Under Asynchronous Sensor Observations
Artificial Intelligence
Summary
Many computer models assume they get new data from sensors all at once, but real sensors often send updates at different times. The authors show that simply holding onto the last reading until a new one arrives, combined with telling the model how long it has been since each sensor updated and when the next update will come, can help make better predictions. They tested this idea on tasks where the timing of sensor updates affects the system's behavior and found that knowing the update schedule is especially helpful when those updates actually influence what happens next. This research helps understand when and how irregular sensor data timing matters for predictions in real-world systems.
world modelasynchronous sensorszero-order holdtime-to-refreshstalenesstransformer modelcontinuous controlmodel-predictive planningsampling schedulerefresh events
Authors
Akash Anand, Abhay Anand, Yash Vishe
Abstract
Learned world models typically assume that observations arrive synchronously, an abstraction inherited from simulators that return a complete state vector at each environment step. Physical sensing instead operates at heterogeneous rates, leaving most observation channels stale at any given instant. Interpolating stale channels introduces measurements that were never observed, while downsampling to the slowest sensor discards valid measurements. A natural alternative is to zero-order-hold the most recent reading and provide the known sampling schedule to the model through two features, staleness and time-to-refresh. We test this prediction using transformer world models across three regimes of increasing causal coupling: open-loop rollouts in continuous-control locomotion, closed-loop model-predictive planning in which each learned model serves as the planner dynamics, and a linear latched-actuator system in which refresh events apply a zero-order-held command to the plant. Our findings show that the effectiveness of time-to-refresh depends on the causal role of the sampling schedule, specifically when refresh events affect the system rather than merely report its state. These results establish when sampling schedules provide useful information for predictive world models operating under asynchronous physical observations.