Action jets speed up robot movement prediction by learning local action effects
Action Jets: Learning the Local Shape of the Action-conditioned Data Manifold
Computer Vision and Pattern Recognition
Summary
Predicting what will happen when a robot moves or acts can take a lot of computing power, making it hard to check many options quickly. The authors show that when looking just a short time ahead, this prediction only needs local information near the current state. They create a tool called an Action Jet that learns how small changes in actions locally warp the possible outcomes, capturing this with a mathematical approximation. This lets planners quickly score many possible actions without repeatedly doing heavy computations. Their method is faster and more accurate than previous models for short-term predictions in robot manipulation tasks.
What this means in practice
- •For robotic manipulation engineers: Rapidly evaluate many candidate robot actions at short horizons using local action effects without costly full simulations.
- •For computer vision developers: Generate accurate, short-term novel viewpoints of scenes by efficiently approximating frame changes from small action steps.
Authors
Anton Arapin, Chahyon Ku, Bernadette Bucher
Abstract
World models are a powerful means of reasoning and planning, but many of these action-conditioned predictive models are high dimensional and computationally expensive. These challenges limit the number of action candidates that can be evaluated by a planner. We show that this cost is avoidable for planners over short horizons, which only require local information with respect to the current state. For a fixed state, the frames reachable by varying a low-dimensional action (a camera movement or an end-effector command) lie on a low-dimensional manifold. We estimate the local shape of that manifold directly via a network trained to estimate how each image region moves per unit of each action axis supervised by data collected from samples by the generator. A correction head fit to trajectory data estimates the shape of this manifold region up to a third-order Taylor polynomial. We call this network estimating the Taylor polynomial describing the manifold shape an Action Jet and compute its radius of validity in closed form. By traversing this approximation of the local manifold shape, we only have to perform inference once per state for any number of candidate actions. Scoring sixteen candidates from a single extraction is over 300x faster than with either source generator, and this gap widens with the number of candidates a planner evaluates. We measure against state-of-the-art action-conditioned generators, IRASim and WorldGym on BridgeData V2, a real 7-DoF manipulation benchmark, and FrameCrafter on DL3DV scenes, where we chart from the novel-view generators DepthSplat and LagerNVS. Because only the tangent comes from the generator and every other term from the data, the Action Jet also predicts held-out frames more accurately than IRASim, the generator that supplied its tangent, at short horizons and at over 100x less compute.