Prediction error on key states predicts planning success better
Not All Errors Matter: Decision-Relevant Prediction Error Predicts Planning Quality
Machine Learning
Summary
Sometimes, predicting everything correctly isn't as important as predicting the parts that affect decisions. The authors show that models with similar overall mistake rates can perform very differently if their errors happen in different areas. They introduce a new way to measure errors that focuses only on the important parts that influence decision-making, which better predicts how well a model plans. They also found that which errors matter most changes depending on the task and that errors can get worse when thinking ahead more deeply.
What this means in practice
- •For robotic system engineers: Improve planning reliability by focusing model training on prediction errors that truly affect robot decisions.
- •For autonomous vehicle developers: Enhance navigation safety by measuring and reducing prediction errors specifically on critical driving states.
Authors
Linhao Wang, Yiyan Fan, Dongjin Huang
Abstract
World models are typically trained and evaluated by prediction error, assuming that more accurate predictions lead to better decisions. We show that this assumption can fail because models with similar total error can differ substantially in planning performance when their errors occur on different state dimensions. We introduce Decision-Relevant Prediction Error (DRPE), which measures prediction error on the state dimensions that affect decisions. We also develop an iso-error evaluation protocol that varies error allocation while keeping total error fixed. In a factored gridworld with known state relevance and a standardized planner, we evaluate 55 controlled and learned models across different error levels and allocations. Total prediction error is weakly related to planning success (Spearman $ρ=-0.25$), whereas DRPE is strongly predictive ($ρ=-0.84$; $-0.98$ within the controlled family). Models with only a 1\% difference in total error can differ by 60 percentage points in planning success (97\% vs 37\%). The relevant error also depends on the task, with model rankings reversing across tasks at the same total error. Deeper imagination further amplifies decision-relevant errors, while learned models exhibit systematic bias on rare but decision-critical events. We formalize sufficient conditions under which DRPE correctly ranks models and total prediction error cannot.