Embodied foundation models unified to improve robot actions
Unifying Embodied Foundation Models via a Model-to-Policy Formulation
RoboticsArtificial IntelligenceMachine Learning
Summary
Robots need to understand what they see and decide how to act. Different robot models do this in various ways, making it hard to connect or improve them together. The authors propose a new way to describe and unify these robot models so they all work from a common idea. This helps combine learning methods and better adapt robots to new tasks. They tested their idea on two existing robot models and showed how it applies to different kinds of robot action planning.
What this means in practice
- •For robot control engineers: Design robot control systems that flexibly combine pretraining and on-the-fly adaptation using a unified model-to-policy framework.
- •For autonomous vehicle developers: Improve decision-making models by integrating perception and action planning under a common probabilistic approach.
Authors
Xinyi Hong, Mingyang Ren
Abstract
Vision-language-action (VLA) models and world-action models (WAMs) have emerged as promising approaches to robot learning and control. These models integrate multimodal perception and action generation, but differ in their architectures, output representations, and learning objectives. In this paper, we present a unified, architecture-agnostic probabilistic formulation for embodied foundation models based on stochastic kernels. Our formulation has two key components: (i) we characterize embodied foundation models through conditional output distributions and action-realization kernels, which together induce task-conditioned robot policies; and (ii) we connect pretraining and post-training to policy deployment, accommodating model-specific pretraining objectives and offline or online post-training strategies. We instantiate the formulation for OpenVLA and DreamZero, covering autoregressive action-token prediction and joint video-action generation.