Unified Video-Action Joint Denoising for Dexterous Action and Data Generation
2026-06-02 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors present Donk, a model that connects videos of hand movements with the actual hand actions needed to perform those movements. Unlike previous models that focused only on predicting future actions based on current observations, Donk models both videos and hand trajectories together, allowing it to generate realistic videos and corresponding hand movements. It can create these predictions based on language, an initial image, and the starting hand position, or even just text. The authors show that Donk performs well in creating accurate hand trajectories, high-quality videos, and smooth action sequences all within one system.
video foundation modelsrobot action policydenoising modelMANO hand modelbimanual trajectoriesvideo-action alignmentconditional samplingtext-conditioned generationdexterous manipulationtrajectory accuracy
Authors
Dingrui Wang, YuAn Wang, Jinkun Liu, Yue Zhang, Mattia Piccinini, Yu Sun, Johannes Betz
Abstract
Recent world action models leverage video foundation models by aligning broad visual-dynamics priors with executable robot actions. We revisit this alignment from a distributional perspective. Existing formulations typically narrow the aligned prior into an observation-conditioned policy distribution over future actions. In contrast, we keep the distribution broader by modeling the joint space of interaction videos and executable hand trajectories under multiple conditioning regimes. We propose Donk, a unified video-action denoising model for dexterous hands. With language, an initial image, and the initial hand state, Donk samples future videos and bimanual MANO trajectories as an action policy. Without the image condition, the same denoising architecture samples paired video-action rollouts from a text-conditioned distribution, turning the aligned video prior into a data engine. Across action, video, and text-only generation evaluations, Donk improves dexterous trajectory accuracy, preserves strong video fidelity, and produces smooth text-conditioned action rollouts under the same unified training recipe.