RynnUnify enables robots to plan and act through video understanding
RynnUnify: Towards Unified Embodied Brain
Robotics
Summary
Robots need to understand their surroundings and plan actions over time to complete tasks. The authors introduce RynnUnify, a large model that learns to predict and reason about videos and language together, helping it understand physical spaces and anticipate future events. They added training methods letting the model decide when to use video predictions for better decision-making and to directly control robots. Tests show RynnUnify improves the ability to plan visually and helps robots succeed in real-world tasks. This work helps bridge understanding and action in embodied AI systems.
What this means in practice
- •For roboticists: Use RynnUnify to improve robots’ abilities to predict and plan over long sequences for complex manipulation tasks.
- •For autonomous vehicle engineers: Incorporate video-based embodied reasoning to enhance understanding of dynamic environments for safer navigation decisions.
Authors
Minghao Zhu, Bohan Hou, Kun Xiang, Zixuan Wang, Yinzhou Tang, Yifei Yang, Zhikai Wang, Tianyi Zhang, Kehan Li, Haoyu Zhao, Siteng Huang, Leigang Qu, Sicong Leng, Dongchi Huang, Mingxiu Chen, Shijian Lu, Yue Wang, Xiaodan Liang, Zhenyu Zeng, Deli Zhao, Chengju Liu, Jianfei Yang, Xin Li
Abstract
We introduce RynnUnify, a 16B unified embodied model that combines strong embodied understanding with long-horizon visual planning through interleaved video--language reasoning. With capabilities spanning spatial reasoning, physical prediction, and grounding, RynnUnify connects an understanding of the physical world with visual anticipation of how it evolves across successive subtasks. Building on these capabilities, we further develop an agentic extension through reinforcement learning to selectively use video predictions for reasoning about unobserved states and interaction outcomes. We also develop a world-action extension of the full RynnUnify model, translating its understanding and prediction capabilities into robot control. To evaluate these capabilities, we introduce Rynn-VP and Rynn-GEU for long-horizon visual planning and anticipatory reasoning. Experiments show that RynnUnify achieves the highest average embodied understanding score among the evaluated unified multimodal models and improves the Rynn-VP overall score by 9.75 points over Cosmos 3 Nano. Its agentic extension improves the Rynn-GEU average by 6.0 points over direct reasoning, while the world-action extension achieves 51.5\% success on 20 selected composite RoboCasa365 tasks and 46.7\% average success across three real-robot tasks. Together, these results establish RynnUnify as a shared foundation for embodied understanding, visual planning, and robot control, taking a step toward a unified embodied brain.