Cross-entropy method models improve action selection in robotics control
In CEM, a World Model Is Also a Proposal Mechanism
Machine LearningRobotics
Summary
The paper studies how a popular search method called the cross-entropy method (CEM) uses predictions from learned world models to pick good actions for robots. The authors find that prediction errors can affect both the current decisions and the future choices considered. They test different models on simulated walking and running robots, showing that interacting with real results helps improve final decisions. Swapping model-based rankings with actual environment feedback once leads to better outcomes. This work helps understand how prediction quality influences action planning.
What this means in practice
- •For robotics engineers: Improve robot control by combining model predictions with real feedback to select better action sequences during planning.
- •For reinforcement learning developers: Refine action search algorithms by updating proposals with environment-evaluated outcomes to reduce error impact in model-based methods.
Authors
Oliver Obst, Frieder Stolzenburg
Abstract
The cross-entropy method (CEM) uses world-model scores to select action sequences and fit the distribution sampled in its next iteration. A scoring error can therefore change both the present decision and the candidates considered later. We evaluate these two roles separately. Four types of predictive model generate CEM traces, and every model rescores every saved candidate pool. Executing the same candidates in the environment provides a reference elite set and proposal update. Across twelve independently trained task-seed units on Walker and Cheetah, the pre-specified proposal distance falls from the first to the final CEM iteration in every unit. Proposal widths contract and fitted means separate relative to the remaining search width. Pairwise ranking agreement stays near chance on Walker and declines on Cheetah; elite-set agreement does not improve. This comparison shows greater variation between scorers than between pool sources on Cheetah; Walker has variation in both and in their pairings. We use the original six units to select Random nonlinear for a one-update intervention, without inspecting intervention outcomes. Replacing its first model-ranked update with an environment-ranked update lowers final realised selected-sequence cost in those six units and in six further units held out from the selection.