Papers for

immersive media developers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

FOAdapter enhances low-bitrate spatial speech coding for immersive audio

FOAdapter: A Plug-and-Play Spatial Tokenizer for Low-Bitrate FOA Speech Coding

Abstract: Spatially encoded speech conveys not only linguistic information but also spatial cues. This enables listeners to distinguish and localize speakers, contributing to more natural and realistic auditory experiences in applications such as remote conferencing and immersive media. Neural speech codecs (NSCs) enable high-quality reconstruction of monaural speech at increasingly low bitrates while preserving linguistic information, but their ability to effectively preserve spatial cues remains limited. We propose FOAdapter, a plug-and-play spatial tokenizer that leverages a pretrained monaural speech codec to efficiently represent both linguistic and spatial information in first-order ambisonic (FOA) speech at low bitrates. Comprehensive evaluation demonstrates that FOAdapter consistently preserves spatial information across all evaluated codecs. Source code and audio examples are available on our demo website.

Mon 12 OctSound
The gist
Speech audio doesn’t just carry what people say but also where they are. This spatial information helps listeners tell speakers apart and feel like they’re in a real place. Existing speech coding methods work well for single-channel speech but lose these spatial details at low data rates. The authors propose FOAdapter, a tool that fits into current speech codecs and keeps these spatial cues intact even when compressing audio to very small sizes.
Open → 2610.16202v1

Open source platform enables low cost 360 video eye tracking crowdsourcing

OpenSAL360: Open-Source Crowdsourcing Platform for Omnidirectional Video Saliency Collection

Abstract: Omnidirectional video saliency prediction plays an important role in many immersive multimedia applications, including viewport-adaptive streaming and compression, foveated rendering, mesh simplification, perceptual quality assessment. Yet progress in this area remains constrained by the cost and complexity of collecting eye-tracking data with VR headsets, which makes large-scale dataset creation difficult to extend. We present OpenSAL360, the first open-source platform for scalable, low-cost 360° video saliency collection. Unlike conventional VR-based protocols, it requires only a standard screen, mouse, and internet connection, enabling parallel saliency data collection from common crowdsourcing assessors without specialized hardware. We validate our collection protocol against seven well-established VR eye-tracking datasets and conduct ablation studies on key interface, pre-, and post-processing parameters. To demonstrate the effectiveness and scalability of the proposed methodology, we collect and publicly release a saliency dataset covering 500 omnidirectional videos annotated by 2,000+ crowdsourcing assessors, making it, to the best of our knowledge, the largest dataset in this field. We make OpenSAL360 publicly available at https://github.com/msu-video-group/OpenSAL360.

Fri 18 SeptComputer Vision and Pattern Recognition
The gist
Collecting where people look in virtual reality videos usually needs expensive VR headsets, making large studies hard to do. The authors designed OpenSAL360, a free tool that uses ordinary computer screens and a mouse to gather this information from many online participants cheaply. They checked that this new approach matches data from traditional VR eye tracking and used it to collect the largest 360° video eye-tracking dataset so far. This makes it easier to study and improve immersive video experiences without costly equipment.
Open → 2609.21480v1