Neural operator improves 3D spatial audio personalization speed and accuracy
HRTF Upsampling and Personalization via Neural Operators
SoundMachine Learning
Summary
Getting accurate 3D sound effects personalized to a person's ears usually requires many measurements. The authors frame this task as learning a function that fills in missing audio data from few samples. They created a method called SPARO that uses machine learning to quickly predict detailed sound patterns from just a few samples, making the process much faster and more precise than previous methods. SPARO works well on different datasets and speeds up sound rendering by 15 to 373 times compared to earlier approaches.
What this means in practice
- •For audio engineers: Create highly customized 3D audio filters from a few measurements to improve virtual reality sound experiences.
- •For game developers: Generate fast and accurate personalized binaural audio spatialization to enhance immersive gaming.
Authors
Meenakshi Krishnan, You Zhang, Ramani Duraiswami, C. Phillip Brown
Abstract
Head-related transfer function (HRTF) spatial upsampling from sparse measurements is a fundamental challenge in personalized binaural audio. We formulate HRTF upsampling as an operator learning problem and introduce SPARO, a neural operator that maps sparse measurements at a set of spatial directions to the HRTF field, a continuous representation that can be queried at arbitrary positions. SPARO projects the measurements onto a learned set of spherical latent modes, processes the latent modes with Transformer blocks, and decodes the resulting representations at query directions. On SONICOM, using the LAP24 measurement configuration, SPARO attains the best log-spectral distortion (LSD) and interaural level difference (ILD) error at every sparsity level from 3 to 100 measurements (18% lower LSD and 43% lower ILD error than the strongest baseline at 100 measurements) while remaining competitive in interaural time difference (ITD) error. It reconstructs a dense HRTF 15 to 373 times faster than the leading neural-field method RANF. We further validate the approach on HUTUBS, demonstrating its effectiveness across HRTF datasets.