PLACE improves binaural audio generation from text video and audio prompts
PLACE: Positional Latent Adaptation via Conditioned Embeddings for Binaural Audio Generation
SoundMultimedia
Summary
Creating realistic 3D sound from text or video is tricky because it needs to simulate how we hear sounds around us. The authors introduced PLACE, a method that extends a previous model to better handle sound direction and space cues using text, video, and optional audio inputs together. They trained their model to pick up on subtle differences in sound timing and volume that our ears use to locate where sounds come from. Tests show PLACE produces clearer and more accurate spatial sounds than previous methods, and listeners prefer its results in many cases.
What this means in practice
- •For game audio developers: Generate spatially consistent binaural sounds from game text or video scenes, enhancing player immersion with realistic audio cues.
- •For augmented reality app makers: Create adaptive binaural audio experiences from mixed text video and audio inputs to improve realism in AR applications.
Authors
Tiernon Riesenmy, You Zhang, Gautam Bhattacharya, Andrea Fanelli
Abstract
We present PLACE, a method that extends the pretrained any-to-audio model AudioX for binaural generation from arbitrary combinations of text, video, and optional audio prompts. PLACE augments video conditioning with Perception Encoder Core features, aligns text and video representations to derive spatial cues, and applies a conditioning-dependent low-rank transformation to the generated latent. The adapter is supervised via decoded-audio interaural level and time difference objectives. Trained on MRSAudio, PLACE improves most metrics over ViSAGe on FAIR-Play and achieves an improved SpatialCLAP score over SpatialSonic on the BEWO-1M Single Static test split. Listener evaluations favor PLACE for video-to-audio and out-of-distribution text-to-audio generation, demonstrating flexible multimodal control and improved spatial consistency.