Unified model improves robot control by linking video action and language
UniWALM: Unified World Action Language Model for Embodied Robot Control
RoboticsArtificial IntelligenceComputer Vision and Pattern Recognition
Summary
Robots usually learn how to act using videos and other signals separately, making it hard for them to plan and remember tasks well. The authors created a single model called UniWALM that mixes videos, robot actions, and language so the robot can use all these things together. This lets the robot describe what it’s doing in words, remember that description, and improve its actions without separate planning steps. The new approach helps robots perform better on tasks like picking and placing objects and finding items, even in real-world tests.
What this means in practice
- •For robotics engineers: Enable robots to perform complex pick-and-place tasks with improved success by jointly using video, action, and language signals in one model.
- •For automation system integrators: Improve warehouse item-finding robots by using textual memory generation to increase success rates without extra hardware or separate planning modules.
Authors
Rahul Sajnani, Diego Romeres, Mingyuan Zhang, Kai Gao, Carl Qi, Baichuan Huang, Srinath Sridhar, Dylan Randle, Lillian J. Ratliff, Jose Barreiros, Manikantan Nambi, Jeroen Vanbaar
Abstract
Motivated by the human ability to integrate experience across language, perception, and action, we introduce UniWALM, a unified world-action-language model that learns video, action, and language in a single shared representation, where each modality is both a conditioning signal and a supervision target. We ask: can jointly generating language enrich this shared representation, improving manipulation and supporting memory-dependent behavior? Unlike prior approaches that attach a reasoning or vision-language model to a video-action backbone and run them as distinct stages, UniWALM unifies all three modalities inside a single mixture-of-transformers backbone whose per-modality diffusion heads co-denoise synchronously, decoding subtask and textual memory in the same pass as action with no separate planning stage; the generated language is re-consumed on the next step, closing a language-as-memory loop without an external planner or memory bank. To keep the action stream reliable under this richer objective, we further introduce Flow Matching with Correction (FMC), a training objective that teaches recovery from perturbed action trajectories at no inference-time cost. UniWALM consistently improves over same-backbone baselines (e.g., FastWAM) on compositional pick-and-place (MolmoSpaces) and across the memory-dependent subset of RMBench, where it reaches 34.5% without any pretrained language or reasoning model and exceeds Mem-0's keyframe-equipped setup (28.5%) using only a single anchor frame. On a real-world Find-the-Item task, generating textual memory lifts success from 11.1% to 61.1%, while state-of-the-art VLAs stay at or below chance (<=33.3%).