Mt wam improves robot action success by predicting motion and visuals
MT-WAM: Reorienting the One-Pass Predictive Representation Toward Action Generation
Computer Vision and Pattern RecognitionRobotics
Summary
Getting robots to act correctly by watching video is tricky because predicting future video isn't always helpful for control. The authors improved a previous method by teaching the system to predict future movement paths and visual features separately, helping it focus better on what matters for making decisions. This change makes the robot's understanding of its environment more useful, enabling it to plan actions more successfully. The approach works well on multiple robot control tasks without needing extra training.
What this means in practice
- •For robotic system developers: Improve robot control by integrating motion and visual predictions to boost task success under dynamic visual conditions.
- •For automation technology engineers: Enhance motion planning in automated systems by adding targeted future motion paths and visual context predictions for better decision-making.
Authors
Yiguang Yang, Jiankun Peng, Xiaoming Wang, Yiran Zhang, Zhibo Fang
Abstract
Fast-WAM shows that video-action co-training improves control without generating future video at inference, making the representation from a single video diffusion Transformer forward central to action generation. However, future-observation prediction does not explicitly prioritize the future dynamics and visual structure needed for control. We present MT-WAM, which retains the original training objectives and adds complementary supervision for future two-dimensional point trajectories and visual features. A lightweight dual-stream branch copied from the video backbone's final blocks provides target-specific processing, while a structured attention mask prevents cross-stream attention. Motion-stream tokens supply additional dynamics conditions to the action expert. Future visual-feature prediction provides supervision in a feature space that captures object and spatial structure. This supervision trains the video backbone to provide more informative visual context for action generation under changing visual conditions, without adding visual-feature-stream tokens to action conditioning. At inference, MT-WAM uses video and motion caches computed once per replan and skips future-video prediction. Without additional embodied policy pretraining, MT-WAM achieves 98.2% success on LIBERO and 73.7% on LIBERO-Plus, exceeding Fast-WAM by 23.8 percentage points on the latter. On RoboTwin 2.0 Clean2Rand, Random success increases from 6.30% to 19.40%; across four real-world tasks, average success increases from 67.0% to 77.8%.