World models improve with sensor timing in asynchronous observations
World Models Under Asynchronous Sensor Observations
Artificial Intelligence
Summary
When machines try to understand the world, they usually expect all their sensors to send data at the same time. But in real life, sensors send information at different speeds, so some data gets old while waiting. The authors studied a way to handle this by telling the model how old each piece of data is and when new data will arrive. They found this helps predict better when the timing of sensor updates actually affects what’s happening, not just when observing. This helps machines learn from and plan with more realistic, mixed-speed sensor inputs.
What this means in practice
- •For robotics engineers: Design robot controllers that handle sensors updating at different rates without losing important data for better real-time decision making.
- •For industrial automation teams: Improve systems controlling machines with mixed-speed sensors by leveraging timing information to enhance predictive control accuracy.
Authors
Akash Anand, Abhay Anand, Yash Vishe
Abstract
Learned world models typically assume that observations arrive synchronously, an abstraction inherited from simulators that return a complete state vector at each environment step. Physical sensing instead operates at heterogeneous rates, leaving most observation channels stale at any given instant. Interpolating stale channels introduces measurements that were never observed, while downsampling to the slowest sensor discards valid measurements. A natural alternative is to zero-order-hold the most recent reading and provide the known sampling schedule to the model through two features, staleness and time-to-refresh. We test this prediction using transformer world models across three regimes of increasing causal coupling: open-loop rollouts in continuous-control locomotion, closed-loop model-predictive planning in which each learned model serves as the planner dynamics, and a linear latched-actuator system in which refresh events apply a zero-order-held command to the plant. Our findings show that the effectiveness of time-to-refresh depends on the causal role of the sampling schedule, specifically when refresh events affect the system rather than merely report its state. These results establish when sampling schedules provide useful information for predictive world models operating under asynchronous physical observations.