World action critic model improves prediction and evaluation in policy learning
World Action Critic (WAC) Model: Joint Foresight and Criticism for Policy Learning
Artificial Intelligence
Summary
It is hard for AI systems to learn what would happen if they took different actions they haven't tried before, because they don't have direct experience of those alternatives. The authors introduce a model called the World Action Critic (WAC) that tries many possible actions and learns at the same time to predict the future and evaluate how good each action might be. This helps the AI remember important details to tell which choices are better, even when it lacks direct examples. They tested this system on self-driving car simulations and showed it can make smarter decisions.
What this means in practice
- •For autonomous vehicle engineers: Enable better decision-making in self-driving cars by jointly learning to predict outcomes and evaluate alternative driving actions.
- •For robotics system integrators: Improve robotic control policies by incorporating joint foresight and action quality evaluation without needing direct examples of all possible actions.
Authors
Xiangteng Zhang, Haoyuan Liu, Yang Guan, Jingliang Duan, Yao Lyu, Tao Zhang, Shuai Zhang, Kehua Sheng, Bo Zhang, Shengbo Eben Li
Abstract
World models have shown considerable promise in sequential decision-making by anticipating the consequences of actions and providing predictive foresight for policy learning. However, learning to reason about counterfactual actions remains challenging, since predictive objectives may underemphasize information critical for action evaluation, while unexecuted actions lack observed futures for direct supervision. To address these challenges, we propose the World Action Critic (WAC) Model, which proposes multimodal action candidates and jointly learns candidate-conditioned future prediction and consequence evaluation. In our proposed framework, critic supervision directly shapes intermediate predictive representations during rollout, encouraging them to retain information needed to distinguish action quality. Observed futures supervise prediction for executed actions, while external task-quality targets provide evaluation signals for counterfactual candidates. Through joint optimization, critic signals from these candidates propagate into predictive representations without requiring their observed futures. Our theoretical analysis characterizes critic-guided feature retention and bounds candidate-selection regret. Applied to autonomous driving, WAC achieves 93.4 PDMS on NAVSIM-v1, demonstrating the effectiveness of jointly learning predictive foresight and consequence evaluation.