UniqueShip improves ship sound recognition with better data controls
UniqueShip: Mitigating Data Leakage in Acoustic Ship Classification Benchmark Datasets
SoundComputer Vision and Pattern Recognition
Summary
Recognizing ships by their underwater sounds helps with tracking and identifying vessels but needs lots of good example data. The authors created UniqueShip, a large new dataset with over 4,000 different ships that avoids a problem where the same ships appear both in training and testing, which can fool the computer into just memorizing sounds. They show that having a bigger variety of ships in training helps more than just having longer recordings. They also tested different computer models and found that specific ship features affect recognition difficulty more than just how far the ship is from the sensor.
What this means in practice
- •For marine monitoring teams: Use UniqueShip to train more reliable models for identifying ships from underwater acoustic data with reduced risk of overfitting on known vessels.
- •For security surveillance operators: Improve underwater vessel classification accuracy by focusing training on diverse ship samples instead of longer audio recordings.
Authors
Connor Hashemi, Trevor Stout, Anthony Hoogs, Jason Parham
Abstract
Underwater Acoustic Target Recognition (UATR) of ships is well-suited for machine learning, yet its progress is hindered by the lack of large, diverse, and publicly available labeled datasets. In this work, we introduce UniqueShip, a machine learning-ready benchmark dataset for UATR applications sourced from the open Ocean Networks Canada (ONC) repository. Unlike previous datasets, we explicitly control for "data leakage" between the training and evaluation sets to ensure more reliable and generalizable model evaluation that does not encourage the model to memorize individual ships. We demonstrate that typical, random data partitioning in two prominent UATR datasets leads to falsely optimistic test performance, increasing accuracy by 10-48 percentage points compared to our more careful partitioning. Ablations on UniqueShip further show that doubling the number of unique vessels improves accuracy by 2.4-2.6 percentage points, while doubling total audio duration improves only by 0.8-1.3 points, indicating that vessel diversity should drive dataset curation more than total hours. We provide baselines with convolutional and transformer backbones, and analyze how ship metadata correlates with classification performance, finding that individual vessel characteristics predict classification difficulty far better than distance to the hydrophone alone. Overall, UniqueShip contains 2,460 hours of ship-radiated audio from 4,218 unique vessels (3,437 hours including background). We publish the dataset, code, and easy-to-download splits at uniqueshipdata.org to foster further UATR research.