World model updates measured for utility using counterfactual deployments
Measuring the Value of World-Model Updates: A Counterfactual Utility Protocol for Continual Adaptation
Machine Learning
Summary
When a computer program that predicts how the world works gets new information, it needs to decide if it should update its understanding. The authors introduced a way to test the value of each update by comparing what happens if the model updates versus if it stays the same at the same moment. Their tests on simulated control tasks showed that always updating can sometimes make performance worse. This new method helps decide when updates really improve predictions rather than just reacting to surprises.
What this means in practice
- •For autonomous vehicle developers: Evaluate if real-time updates to driving models improve safety and performance using counterfactual comparisons.
- •For robotics engineers: Assess the benefits of updating robot control policies on the fly by measuring actual impact on task outcomes.
Tested on simulated data.
Authors
Anqi Peter Li, Kaden Kim
Abstract
Continual world models must decide whether new data justify changing the model. Fixed replay schedules and prediction-error triggers specify when to update, but neither reveals the value of an individual update: one deployment run cannot show how the same model would have performed at that moment had it held its parameters. We introduce the fork ledger, which branches a deployment stream at pre-registered decision points into matched update and hold continuations under common random numbers. It evaluates both continuations on the same episodes and records $ΔR = R_{\mathrm{update}} - R_{\mathrm{hold}}$. Always applying one fixed update mechanism lowers return on all three simulated control tasks: CartPole ($-144.0$; checkpoint-bootstrap $95\%$ CI $[-185.4,-116.1]$, against a converged return near $650$), Walker ($-82.8$; $[-101.1,-61.7]$) and Cheetah ($-18.6$; $[-29.0,-6.6]$). Divergence is an outcome of applying the update, so the estimand counts every attempted fork; restricted to the $693$ of $720$ that did not collapse, CartPole and Walker are unchanged in sign ($-113.4$ and $-82.1$) and Cheetah becomes unresolved ($-3.9$; $[-17.5,+13.0]$). The task is the unit of inference: each contributes $240$ attempted forks over five pretrained checkpoints crossed with two drift directions. The ledger makes counterfactual utility observable for a fixed mechanism, allowing triggers to be judged by the updates they select rather than by surprise detection alone.