Papers for

hearing aid designers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Object centric audio representation enhances multiple sound tasks

Self-Supervised Object-Centric Representation for Multi-Task Audio Analysis

Abstract: This paper studies the capability of an object-centric audio representation to support multiple downstream tasks within a unified framework. The object-centric model is trained in a self-supervised setup using slot attention by reconstructing Audio Spectrogram Transformer (AST) patch embeddings from a set of latent slots. It is trained on synthetic audio mixtures, which provides controlled multi-source audio scenes. These slot representations are evaluated for the classification, detection, and separation of sound events by training task specific output layers. On the classification task, the model achieves an F1-score of 0.8, comparable to clip-level AST embeddings which achieve 0.7. For sound event detection, it reaches a macro F1-score of 0.7, outperforming clip-level AST embeddings at 0.4 and matching AST patch embeddings at 0.7, which encode finer time-frequency information. In source separation, it achieves an SI-SDR improvement (SI-SDRi) of 18.4 dB. Furthermore, in a zero-shot source separation setting, the model achieves an SI-SDRi of 14.5 dB. These results demonstrate that slot-based representations capture transferable and compositional structure in audio scenes, enabling generalization across tasks and datasets.

Fri 30 OctSound
The gist
Understanding sounds in a complex audio mix can be tricky. The authors show that representing sounds as separate objects—like pieces in a puzzle—helps computers recognize, detect, and separate different sounds better. They trained their system on fake audio mixes so it could learn how to break down sounds into parts without needing labeled examples. This approach works well across several tasks, suggesting it could generalize to many real-world audio problems.
Open → 2610.41870v1

Wearable devices estimate audio quality and listening effort

Physiology-Informed Audio Quality and Listening Effort Estimation

Abstract: Audio systems aim to maximise the listener's perceived quality, traditionally evaluated through mean opinion scoring (MOS), a subjective process that is slow, costly, and collected only after playback. Machine learning now predicts MOS directly from the audio, but in generalising across participants it tends to discard the subject variance that defines individual perception. Neurophysiological measures may offer a complementary signal taken directly from the listener, objective, implicit, and continuous, though prior work has relied on laboratory-grade EEG and fNIRS. We ask how practical consumer-grade devices compare, and whether they might recover population-level audio-quality trends and, when calibrated per listener, track that listener's effort. Recording 32-channel laboratory EEG alongside wearable EEG, fNIRS, and eye tracking from thirty listeners, our results suggest commercial wearable and optical sensing can decode audio quality at statistically significant levels and recover listening effort per listener. These findings may point towards platforms tuned not only to perceived quality but to the effort a clip demands, read from the listener's own response as they listen.

Tue 27 OctSoundNeural and Evolutionary Computing
The gist
Audio quality is usually judged by people after hearing sounds, which takes time and can vary a lot. The authors show that wearable brain and eye sensors can measure how good the sound is and how hard it is to listen to for each person. They compared lab equipment with consumer devices and found that cheaper, wearable tools still give useful information. This could help make sound systems that adapt to each listener's experience in real time.
Open → 2610.36865v1

Pupil and gaze signals predict lipreading ability better than mouth focus

Predicting Individual Differences in Lipreading From Eye-Tracking and Pupillometry During Silent Visual Speech

Abstract: Lipreading ability varies widely, including among people who are hard of hearing. Although mouth-directed attention is linked to speechreading outcomes, it remains unclear how much lipreading skill can be inferred from viewing behavior and which signals carry this information. We examine whether individual differences in lipreading ability are reflected in viewing behavior during silent speech and quantify the contributions of gaze dynamics, pupil-linked signals, and spatial viewing allocation. We introduce the Lipreading Perception Study (LiPS) dataset and a tracker-independent preprocessing pipeline that synchronizes SR Research EyeLink gaze and pupil recordings with the stimuli. Twenty-four adults (age 32 $\pm$ 10 years) completed a four-alternative forced-choice silent lipreading task. An extreme-groups protocol yielded 18 participants labeled as stronger or weaker lipreaders. Each participant-trial included four aligned views: lip-region images, frame-level gaze, pupil summaries, and spatial viewing proportions. A four-branch neural network with cross-modal attention predicted labels using participant-disjoint splits and systematic ablations. The full model reached accuracy 0.680 $\pm$ 0.040 across five folds, outperforming naive fusion baselines in accuracy. The pupil and viewing-summary branch was most informative: removing it caused the largest degradation (0.618 accuracy, 0.623 AUC), while it alone achieved 0.675 accuracy. The lip-image stream contributed little to participant-level discrimination. Individual differences in lipreading are reflected in pupil-linked and frame-level gaze signals rather than in how much time a viewer spends looking at the mouth. Synchronized gaze and pupillary measures may provide objective markers of visual speech perception, warranting investigation in larger, clinically diverse populations. LiPS provides a resource for extending this work.

Sat 17 OctComputer Vision and Pattern RecognitionHuman-Computer Interaction
The gist
People differ a lot in how well they can understand speech by watching lips. This paper shows that how pupils react and subtle eye movements reveal more about lipreading skill than just looking at the mouth more. The researchers collected eye and pupil data while people silently lipread and used a neural network to find patterns linked to better or worse lipreading. This suggests new ways to measure lipreading ability objectively beyond just tracking where someone looks.
Open → 2610.24898v1

Multi-agent auditory scene analysis improves sound source localization speed

Multi-agent Auditory Scene Analysis: Improved Localization Speed and Robustness by Multi-beamformed Speech Quality Feedback

Abstract: A real-time auditory scene analyzer (ASA) aims to carry out the tasks of locating, separating and classifying the sound sources present in a given acoustic environment. Recently, an effort has been made into modelling an ASA as a multi-agent system, with each one of its agents performing one of the aforementioned tasks and communicating their results to the rest of their peer agents. These communication routes are used as feedback loops to fix local errors at a global level, providing robustness while reducing local complexity. An example of the benefits of this approach is the optimization of speech quality by correcting in real-time the estimated location of the speech source of interest. However, their optimization speed has been shown to be considerably slow. One possible reason is that it solely relies on a series of single quality estimations (provided by a reference-free quality estimator model) that vary considerably from one window to the next, which results in a difficult search space to optimize. In this work, a new optimization mechanism is proposed that instead relies on a series of sets of quality estimations over a range of locations, providing a clearer view of the search space, simplifying its optimization. The proposed ASA now has a considerably smaller optimization time, is more accurate, and is more stable when being evaluated in real-life acoustic scenarios to correct higher levels of localization errors, all while being less complex than previous efforts. The only trade-off is that there is an increase in the response time of the quality estimation agent, but the complete ASA is still able to run in real-time. The performance shown in this work again shows the benefits of modelling an ASA as a multi-agent system.

Wed 30 SeptArtificial IntelligenceMultiagent Systems
The gist
Finding and separating sounds in a noisy place is tricky for computers. The authors worked on a system where different small agents each handle locating, separating, or identifying sounds and share information with each other. They improved how quickly the system corrects errors in locating speech by using multiple quality checks at once instead of one at a time. This makes the system faster, more accurate, and better at dealing with tough real-world noise, all while still working in real time.
Open → 2610.00538v1

Bathroom sound recognition model learns to identify activities from audio

SincDPNet: Interpretable Raw-Waveform Bathroom Activity Recognition for Assistive Living

Abstract: Bathroom acoustic-event recognition can support ambient assisted living in settings where continuous video monitoring is undesirable. However, practical deployment requires models that are compact, interpretable, and robust to changes in the recording environment. This work introduces \dataset{}, a seven-class bathroom acoustic-event dataset containing 21{,}387 annotated clips recorded across five environments, and proposes SincDPNet, a compact raw-waveform classifier with a learnable sinc filter bank followed by a depthwise-separable convolutional body. Each sinc filter is controlled by two frequency parameters, allowing the learned passbands to be inspected directly in hertz while keeping the front end small. To reduce room-specific leakage, recording sessions and environments are separated before overlapping windows are assigned to the training, validation, and test partitions. We further use multi-objective Bayesian optimization as a design tool to examine the validation performance--model-size trade-off across 24 configurations. The selected designs span different operating points: the best-performing model achieves 80.2\% accuracy and 0.760 macro-F1 with 14{,}040 parameters, while the compact $N_f=25$ configuration uses only 2{,}848 parameters and achieves 75.7\% accuracy, 0.661 macro-F1, and 0.716 MCC on the held-out environment. Analysis of the learned filters and confusion patterns shows that spectral overlap contributes to confusion among water-related events, while the \textit{Door}/\textit{Walker/Crutch} errors also reflect similarities in their transient temporal structure.

Mon 28 SeptSoundArtificial IntelligenceMachine Learning
The gist
Listening to sounds in a bathroom can help assistive living by monitoring activities without video. The authors created a new dataset of bathroom sounds and designed a small, easy-to-understand computer program that listens to raw audio and recognizes seven different bathroom activities. Their model uses a special type of filter that can be directly inspected by frequency, making the model more interpretable. They showed their model can work well even when tested in different rooms.
Open → 2609.34907v1

New way to improve audio signals by focusing on sound levels

Prox-Friendly Log-Magnitude Prior on Complex-Valued Signal

Abstract: The logarithmic transform is essential in audio signal processing since human auditory perception is approximately logarithmic with respect to magnitude. However, directly incorporating prior knowledge about signals (e.g., harmonic structure) in the log-magnitude domain into optimization problems solved by standard proximal splitting algorithms remains challenging. To address this issue, this paper proposes a novel regularizer termed EPILOG (Exponential Penalty for Imposing priors on LOG-magnitude). EPILOG indirectly imposes prior knowledge on the log-magnitude of a complex-valued signal through regularization of an auxiliary variable that is shown to be linked with the log-magnitude. Furthermore, we derive its variable-wise proximity operators and develop a proximal splitting algorithm using these operators. Experiments on speech dereverberation demonstrate the effectiveness of the proposed regularizer, particularly in promoting cepstral-domain sparsity.

Mon 28 SeptSound
The gist
Audio sounds are naturally understood by humans through changes in loudness on a logarithmic scale. The authors found it hard to directly use this loudness information when improving audio using existing math tools. They introduced a new technique called EPILOG that helps apply prior knowledge about sound patterns in the loudness scale indirectly, making the math easier to solve. They showed it works well on improving recorded speech by reducing echoes.
Open → 2609.34445v1

WhisperVC-AV improves noisy whisper to normal voice conversion

WhisperVC-AV: Audio-Visual Content Restoration for Noise-Robust Whisper-to-Normal Voice Conversion

Abstract: Environmental noise obscures the content cues needed for whisper-to-normal conversion. We propose WhisperVC-AV, which restores acoustic content features while leaving the original WhisperVC conversion module unchanged. Its context-guided restoration module combines temporal acoustic context with synchronized lip features through attention and gated residual correction. Experiments on AISHELL6-Whisper show lower character error rates (CERs) than WhisperVC across three ASR systems, on clean speech and under all six signal-to-noise ratio (SNR) conditions with MUSAN noise. The largest gains occur at 0 dB SNR, where Qwen3-ASR CER falls from 36.29% to 28.88%. WhisperVC-AV also improves predicted speech quality while maintaining speaker similarity. The CER gains extend to unseen background noise without retraining, while visual controls support the use of utterance-specific lip cues. Audio examples are available on our demo page.

Sat 26 SeptSound
The gist
Turning whispered speech into normal talking voice is hard when there’s background noise because the sounds are harder to understand. The authors created WhisperVC-AV, which uses both the sound and lip movements to better guess what was said. This method helps computers recognize whispered words more accurately, even with lots of noise, without changing the main conversion process. It also keeps the speaker’s voice sounding natural.
Open → 2609.32843v1

Multimodal dataset improves attention tracking with brain and eye signals

MAESTRO: a Multimodal Auditory-attention Egocentric Speech-TRacking Open corpus

Abstract: Humans rely on gaze, head movements, and visual cues to attend to speakers in noisy environments, yet auditory attention decoding (AAD) has been studied primarily using electroencephalography (EEG). We introduce the Multimodal Auditory-attention Egocentric Speech-TRacking Open (MAESTRO) corpus, the first AAD dataset to simultaneously record EEG, eye gaze, pupillometry, egocentric video, and head inertial measurement unit (IMU) data. MAESTRO includes four competing speakers and background noise across multiple signal-to-noise ratio (SNR) conditions, enabling attention decoding under realistic listening scenarios. Through a four-speaker attention decoding benchmark, we show that combining behavioral and physiological signals improves decoding performance over EEG-only approaches, enabling future advances in multimodal auditory attention decoding. These findings open the door to new applications, analyses, and methodological advances in multimodal AAD. The complete dataset is publicly available at https://huggingface.co/datasets/aspire-osu/maestro-eeg-dataset . The official code repository is available at https://github.com/ASPIRE-OSU/MAESTRO .

Fri 25 SeptHuman-Computer InteractionMachine LearningSound
The gist
People use where they look, head movements, and seeing cues to focus on someone talking in noisy places, but most studies only use brain waves to follow attention. The authors created a new dataset called MAESTRO that records brain activity, eye gaze, pupil size, video from a wearer’s perspective, and head motion all at once with multiple people talking. They found that combining these different signals helps track who a person is listening to better than just using brain waves alone. This dataset and code are publicly available for others to build better listening technology.
Open → 2609.31898v1

Audio-visual method enhances target speaker voice with low delay

Adapting Personalized Speech Enhancement for Low-Latency Audio-Visual Target-Speaker Extraction

Abstract: Online audio-visual target-speaker extraction aims to remove competing voices while preserving speech quality and bounding lookahead. Existing extractors are built and evaluated for separation on synthetic mixtures, leaving listening quality and meeting behavior largely untested. We introduce Audio-Visual Personalized Voice Quality Enhancement (AV-PVQE), which approaches these requirements from the other direction. We start from a personalized speech enhancement model that reconstructs a requested voice at high quality but confuses the target in 46% of two-speaker mixtures despite clean enrollment. Adding mouth features at its speaker-conditioning input and jointly fine-tuning the visual and reconstruction networks reduces this rate to 1.6%, with no future frames and 20 ms of algorithmic delay. Compared with an online autoregressive audio-visual extractor, AV-PVQE yields separation gains on two synthetic benchmarks and larger gains on recorded meetings, and keeps its advantage on excerpts with more speakers than the fine-tuning mixtures. In personalized P.835 listening tests on two meeting corpora, it improves overall quality over this extractor by 0.57 and 0.63 MOS, with similar mean rating relative to the starting model. Preservation and rejection tests show that it keeps the target intact when no competing voice is present and suppresses competing speech when the target is absent.

Thu 24 SeptComputer Vision and Pattern RecognitionSound
The gist
When there are multiple people talking, it can be hard to focus on just one person's voice clearly. The authors designed a system that uses both sound and video (like watching mouth movements) to cleanly extract one speaker's voice from a mix of voices in real time. Their method improves the quality of the chosen speaker's voice and reduces confusion from other voices, doing this quickly with minimal delay. They tested it on real meeting recordings and listening tests, showing better results than previous approaches.
Open → 2609.30631v1

Sparse graph method improves audio-visual speech cleaning efficiency

G-Mamba: Sparse Graph-Guided Mamba for Audio-Visual Speech Enhancement

Abstract: Lightweight audio-visual speech enhancement (AVSE) models face a critical trade-off between computational efficiency and cross-modal alignment accuracy. While simple concatenation lacks relational expressiveness, dense cross-attention incurs computational overhead and is prone to unreliable cross-modal correspondence under strong acoustic interference. We propose Sparse Graph-Guided Mamba (SG-Mamba), a lightweight AVSE framework that integrates a sparse heterogeneous graph with a linear-complexity Mamba backbone. The graph explicitly models modality-specific relations through content-adaptive attention and cross-frame audio-visual connections, while Mamba captures long-range temporal context. We further introduce an audio skip connection to preserve spectral detail without sacrificing noise suppression. Evaluated on LRS3, SG-Mamba achieves competitive or superior performance against strong lightweight baselines and reaches 13.091 dB SI-SDR under noise-only condition. It also remains robust in cluttered multi-speaker conditions with a competitive cost of 3.45 G MACs (or 6.90 G FLOPs). Results on VoxCeleb2 further suggest that explicit structural priors improve robustness, generalizability, and computational efficiency in lightweight AVSE.

Wed 16 SeptComputation and LanguageSound
The gist
Cleaning up speech recorded in noisy places is hard, especially when using both sound and lip movement to understand speech. The authors created a lightweight system called SG-Mamba that uses a smart graph to connect sounds and images in a way that saves computing power while keeping good accuracy. Their method also keeps some original sound details to avoid losing voice quality. Tests show SG-Mamba works well even in tricky multi-speaker noisy settings and runs efficiently on modest hardware.
Open → 2609.18009v1