Ultraworld improves autonomous ultrasound scanning with learned models
UltraWorld: Learning Interactive Ultrasound World Models from Untracked Clinical Videos with Acoustic Sampling Map
Computer Vision and Pattern RecognitionRobotics
Summary
Ultrasound scanning requires moving a probe and interpreting images, which is usually hard to automate because it's difficult to link probe movements to image changes. The authors of this paper developed UltraWorld, a method that learns how probe motions change ultrasound images by training on existing clinical videos without needing detailed motion data. They create synthetic training pairs of probe positions and images, then train a model that predicts future images based on probe moves. This helps robotic systems follow planned scanning actions more accurately, potentially improving autonomous ultrasound exams.
What this means in practice
- •For medical device developers: Improve autonomous ultrasound probes to predict images from movements without requiring costly motion annotations.$Commercial implications: Enables commercial robotic ultrasound systems to learn from routine clinical videos, reducing need for expensive pose tracking devices.
- •For robotic surgery teams: Use improved world models to guide ultrasound probe positioning during surgeries with more precise real-time image predictions.
Authors
Keke Yang, Erqi Wang, Sainan Guan, Hongliang Ren
Abstract
World models can enable autonomous ultrasound scanning by predicting the outcomes of probe motions from local observations. Learning this action--observation relationship typically relies on synchronized video--pose pairs, which are costly to collect at scale and largely unavailable in routine clinical recordings. Reliable action following further requires modeling ultrasound's cross-sectional sampling geometry. We present UltraWorld, a self-distillation recipe that transfers priors from clinical ultrasound videos into interactive world models without real action annotations. Starting from clinical videos, we adapt a video foundation model into an ultrasound generator conditioned on reference images and anatomical masks. Anatomical masks sampled along programmable trajectories through 3D anatomy provide spatial guidance for synthesizing action--video pairs. We then use these synthetic pairs to self-distill the generator into a world model that predicts future observations from local observations and actions, without requiring anatomical masks or other 3D assets at inference time. To further improve action following, we introduce the Acoustic Sampling Map (AsMap), which represents probe poses and imaging settings as pixel-wise 3D sampling positions, beam directions, and depths. Experiments demonstrate improved prediction fidelity and action following. Across nine simulated closed-loop local planning episodes, UltraWorld reduces the mean final distance to the goal and orientation error by 29\% and 38\%, respectively, compared with visual servoing. Project Page: https://ultraworld-project.github.io/.