Audio llms learn to alert users proactively in real time
I'll Keep an Ear Out: Teaching AudioLLMs Proactive Audio Assistance
SoundComputation and Language
Summary
Audio large language models usually wait for a user to ask a question before responding. These researchers taught such models to listen continuously and decide on their own when to notify users about important sounds, which helps people like those who are Deaf or Hard of Hearing. They introduced a method called Interrupt and Silent Modeling (ISM) that uses special signals within the model to know when to interrupt or stay silent. Their tests show this method can detect important sounds accurately and quickly, even in noisy environments.
What this means in practice
- •For wearable device developers: Integrate the ISM approach to enable wearables that notify Deaf and Hard of Hearing users about important environmental sounds automatically.$Commercial implications: Allows creation of assistive wearable devices offering real-time proactive audio alerts, improving user safety and awareness.
- •For smart home system engineers: Add proactive audio alert features to smart home devices to warn users about relevant sounds without manual queries.
Authors
Amit Kumar Singh Yadav, Ritvik Shrivastava, Xuan Zhang, Seungwhan Moon, Shashank Jain, Pinar Donmez, Babak Damavandi
Abstract
Audio large language models (AudioLLMs) operate reactively, responding only when queried. We introduce proactive audio assistance, where an AudioLLM monitors an audio stream and autonomously decides when to alert the user from a single natural-language intent, motivated by wearable applications for Deaf and Hard of Hearing users. We propose Interrupt and Silent Modeling (ISM), a model-agnostic paradigm that embeds proactive decisions into LLM decoding via two special tokens: \texttt{<interrupt>} and \texttt{<silent>}, capturing four states: onset detection, sustained-relevance triggering, irrelevance suppression, and de-duplication. Applied to Qwen2-Audio-7B, ISM achieves 99.6\% interrupt F1 and perfect de-duplication recall on ESC-50. On noisy Epic-Sounds kitchen audio, ISM achieves the highest interrupt F1 without domain-specific training, the only method maintaining strong onset detection without over-triggering or over-suppression. Streaming evaluation confirms real-time viability with 3.5-second average latency.