Underwater robot joystick actions paired with video and text
AQUAACT: A Joystick-Level Underwater Teleoperation Dataset and Closed-Loop Benchmark
Robotics
Summary
Controlling underwater robots usually requires a human pilot using a joystick, watching the robot’s camera view to steer it. The authors created AQUAACT, a dataset that pairs underwater video footage with the exact joystick commands pilots use, plus text descriptions. This dataset includes real and simulated underwater missions and helps train AI to understand how to steer underwater robots. Their tests show that AI trained in simulation can help reduce the need for expensive real-world training but fully autonomous underwater control remains a big challenge.
What this means in practice
- •For underwater vehicle operators: Train AI-assisted control systems to reduce pilot workload using paired joystick commands and video data.
- •For robotics software developers: Use AQUAACT dataset to develop and evaluate underwater robot autonomy models that translate camera input into movement commands.
Authors
Zhenqi Wu, Trung Tien Dong, Aditya Penumarti, Mohamed A. Fouad, Menna A. Fouad, Mingyang Mao, Jane Jaejeong Shin, Xiaomin Lin
Abstract
Underwater robots support a growing range of work, from inspecting offshore structures and pipelines to surveying reefs and accompanying divers. Most of these tasks, however, are still flown by hand by a pilot, which raises the question of whether they can be carried out autonomously instead. On land, vision-language-action (VLA) models already let robots carry out tasks autonomously from language instructions, which makes them a natural candidate for taking the pilot's place. Underwater, however, there is little data to train such models. We take inspiration from how these tasks are flown today: the pilot watches the camera image and steers with a joystick, whose commands the flight controller turns into motion. We therefore record the pilots and introduce AQUAACT, to our knowledge the first publicly available language-annotated underwater dataset pairing camera images with human joystick commands. It comprises 1.9 hours of real-world and 17.9 hours of simulated demonstrations, both flown by human pilots, together with the vehicle's inertial measurements. Experiments with AQUAACT show that its simulated demonstrations transfer to real data and greatly reduce the real demonstrations a policy needs. A closed-loop benchmark of seven policies in simulation, from imitation-learning baselines to VLA models, each trained with its default recipe, shows that these tasks remain far from solved: no policy completes more than a fifth of the episodes. Pretraining on land data gives large VLA models no advantage over small policies trained from scratch, a gap that land-trained models have yet to close underwater.