ME-VLM unifies vision language models for physical and agent tasks
ME-VLM:A Unified VLM for Embodied Cognition and Agent Coordination
Computer Vision and Pattern Recognition
Summary
Understanding and interacting with the physical world is hard for AI because it needs to connect what it sees and hears with real-world environments, rules, and feedback. The authors created ME-VLM, a vision-language model designed to help AI better perceive, plan, and act in both digital and physical spaces. They trained it using diverse tasks that involve observing actions and outcomes to improve decisions. ME-VLM performs well on tasks like navigation, driving, and agent coordination, and it can run efficiently on devices with limited computing power.
What this means in practice
- •For robotics engineers: Develop robots that better understand their surroundings and plan actions combining vision and language tasks in real-world environments.
- •For autonomous vehicle teams: Enhance driving systems with models that integrate visual and verbal cues for improved navigation and decision-making in complex scenarios.
- •For mobile device developers: Implement efficient on-device vision-language models allowing AI tasks like navigation and interaction to run faster with less latency.$Commercial implications: Enables sale of responsive AI features on edge devices by compressing vision-language models for efficient local inference.
Authors
Foundation Model, Li Auto Inc
Abstract
Physical AI requires models to ground visual and linguistic understanding in real-world environments while accounting for environmental constraints and execution feedback. We introduce MachEmbodied-VLM (ME-VLM), a unified vision-language model with two variants, 4B and 35B-A3B, that brings together embodied cognition and multimodal agent capabilities. Our work emphasizes physical perception and spatiotemporal reasoning, together with planning, interaction, and outcome assessment in both digital and physical environments. We construct training data spanning embodied and multimodal agent tasks, including execution observations and feedback to support outcome assessment and decision refinement. The training pipeline comprises embodied capability injection, separate reinforcement learning of embodied and multimodal-agent experts, and multi-teacher on-policy distillation that consolidates their complementary capabilities into a single model. Experiments show competitive performance on both embodied and agent benchmarks, as well as on autonomous-driving and embodied-navigation tasks. For edge deployment, visual token compression, W4A8 quantization, and hardware--software co-optimization enable on-device inference of the 4B variant on the M100, reducing prefill latency from 400 ms to 188 ms. Project Page: https://machembodied.com/ME-Brain/ME-VLM.html Code Repository: https://github.com/MachEmbodied/ME-VLM