Predicting multiple hidden vehicles by listening to traffic sounds
Predictive audio representations for early detection and tracking of hidden dynamic objects
SoundArtificial Intelligence
Summary
Detecting hidden vehicles early is important for safety, especially for autonomous cars. The authors created a system that listens to traffic sounds and can tell how many vehicles are nearby, what type they are, and where they are coming from, even when they are out of sight. Their system learns by predicting future audio patterns from past sounds without needing labels, then is fine-tuned to identify multiple vehicles. They also collected a new dataset with multiple hidden vehicles to test their system, which performed better than previous methods and worked well in new driving scenarios.
What this means in practice
- •For autonomous vehicle engineers: Enable early detection and classification of multiple hidden vehicles using audio signals to improve safety and planning.
- •For urban traffic monitoring teams: Monitor traffic flow including vehicles hidden by infrastructure by analyzing ambient road sounds to better understand traffic dynamics.
Authors
Katerina Vinciguerra, Moritz Brandes, Danilo Hollosi, Letizia Marchegiani
Abstract
Predicting potential dangers is core to safety. Forecasting the presence of other traffic agents is core to danger prediction. Occluded traffic agents challenge detection systems as they might become visible too late, leaving the autonomous vehicle too little time to identify, plan and act accordingly in a robust and safe way. Previous works proved that auditory perception, being omnidirectional and not constrained by a field-of-view, provides fundamental cues for early spotting of different road users, even when hidden by other vehicles or infrastructures. Yet, those contributions deal with scenarios with only one vehicle present, and they either identify the type of vehicle or estimate its direction of arrival. In this work, we move forward, and propose a multi-task system which, simultaneously, estimates the number of vehicles present, their type, and their direction of arrival. Our methodological contribution is a two-stage pipeline: a self-supervised pre-training stage inspired by the Joint- Embedding Predictive Architecture (JEPA) applied directly to multichannel raw waveforms, followed by supervised multi-task fine-tuning with a bidirectional LSTM and three classification heads. The pre-training stage trains the encoder without labels, pushing it to predict the latent representation of a future audio segment from its past context. To train and test our framework, since no suitable dataset was publicly available, we collected an ad-hoc one covering Non-Line-Of-Sight scenarios with multiple traffic agents simultaneously operating. Experimental evaluation shows that our method outperforms the state of the art; it also proves that our design choice allows the model to learn robust representations, which can be transferred to an unseen driving scenario, maintaining reasonable and stable performance.