Asynchronous Multimodal Diffusion Policy Composition via Latency-Aware Guidance Fusion
RoboticsArtificial Intelligence
Summary
The authors studied how robots can better learn tasks by combining different types of sensor information that arrive at different speeds. They found that current methods either slow down fast sensors or struggle to add new sensor types easily. To fix this, they created LAG-Fusion, a way for each sensor type to work at its own speed and still combine their information smoothly. They tested this in tasks where robots use both vision and touch sensors and showed that their method helps robots respond faster and perform better than older methods.
Authors
Zihao He, Hongjie Fang, Shirun Tang, Cewu Lu, Haoshu Fang
Abstract
Diffusion policies have shown strong potential for robotic imitation learning, and recent extensions incorporate additional modalities to improve manipulation performance. However, these modalities often differ not only in information content but also in sensing rates and inference latencies. Existing multimodal diffusion policies typically rely on synchronous fusion or manually designed multi-frequency architectures, which either slow down high-frequency feedback or limit extensibility to new modality combinations. We propose LAG-Fusion, a latency-aware guidance fusion framework for asynchronous multimodal diffusion policy composition. LAG-Fusion allows modality-specific policies to operate at their native inference rates and contribute denoising guidance whenever available. To make asynchronous composition consistent, we derive a reference-frame rebasing rule for diffusion variables under relative action representations, enabling delayed guidance to be aligned before fusion. We instantiate LAG-Fusion in contact-rich manipulation by composing a low-frequency vision policy with a high-frequency force policy. Experiments under heterogeneous modality latencies show that LAG-Fusion improves policy responsiveness and task performance over synchronous fusion and specially designed force-aware baselines.