Learning from World Feedback: Why Model Uncertainty Fails as a Risk Signal in Model-Based RL
Machine LearningArtificial Intelligence
Summary
The authors argue that learning signals for control tasks should be based on actual feedback from the environment, not just internal guesses from model predictions. They tested this by comparing standard uncertainty measures inside models to real-world signals like lidar distance and time-to-collision, finding that real-world feedback better predicts and reduces collisions. Their work shows that uncertainty inside a model does not reliably indicate task risk, so they suggest using direct outcome signals, verifying any proxy signals before use, and training feedback models on actual results when direct feedback isn't available. These ideas can improve safety not just in model-based control but also in other settings like language model alignment.
Authors
Zhaohui Wang
Abstract
The RLxF programme argues that learning signals should come from world feedback rather than from internal model proxies. We instantiate this position in safe model-based control and distil it into three concrete design principles. Empirically, across four world-model architectures spanning a 2x MSE range, MPC planning is statistically equivalent (TOST, n=200), and dynamics-based uncertainty penalties increase collision rates from 26% to 34%: the standard MBRL safety proxy is anti-correlated with safety in this regime. Replacing the model-internal proxy with three world-feedback signals (a sensor-derived margin via minimum lidar, a temporal signal via time-to-collision, and an outcome-supervised feedback model g_psi trained on prior collision labels, structurally analogous to outcome-trained reward models in RLHF) reduces collisions to 1-14% without retraining the world model or the planner. The mechanism is structural: model uncertainty has support over state-prediction space, whereas task risk has support over constraint boundaries, with empirical correlation r < 0.15. From this we extract three RLxF principles (ground risk in world outcomes, validate proxies before deployment, and substitute outcome-trained feedback models when direct world signals are unavailable) and argue they apply equally to model-based control and to verifier-based or RLHF approaches in LLM alignment.