Cast-asd enables fast and accurate detection of active speakers online
CAST-ASD: Causal Attention over Speakers and Time for Online Active Speaker Detection
Computer Vision and Pattern RecognitionSound
Summary
Detecting who is speaking in a video with many faces is hard, especially when decisions must be made quickly without seeing the whole clip. The authors created CAST-ASD, a method that looks at all faces in a scene at once while only using past and current information, so it works in real time. This design helps reduce errors where the system confuses one speaker for another, especially when many people are visible. CAST-ASD is efficient, using fewer parameters than some previous methods, and performs better on challenging datasets.
What this means in practice
- •For video conferencing developers: Improve real-time speaker tracking across multiple participants without delays or confusion between speakers.$Commercial implications: Enables creation of smarter video conferencing tools that accurately identify who is speaking live, enhancing user experience and meeting commercial demand.
- •For broadcast production teams: Automatically detect the active speaker in live multi-person broadcasts for more accurate camera switching and captioning.
Authors
Pratham Gupta, Haotian Qi, Tushar Sandhan, Gabriel Skantze
Abstract
We present CAST-ASD (Causal Attention over Speakers and Time), a causal joint model for active speaker detection (ASD). Recent ASD models improve performance by jointly processing all faces in a scene, but operate offline, with the full clip available before any frame is labelled. Low-latency causal detectors instead process each face independently. CAST-ASD combines joint face reasoning with fully causal inference. It applies scene-level attention across faces within the current frame and causal temporal attention over each face's history, enabling information sharing across speakers with zero lookahead. Its parameter count does not grow with the number of faces. On MSDWild, where most negative face-frames contain a visible competing speaker, CAST-ASD reduces competing-speaker false alarms by 32% relative to the causal per-face detector it extends, with larger gains as scene occupancy increases. It outperforms all causal baselines on both validation splits while using only one-tenth the inter-speaker parameters of a three-block fixed-roster convolution. Code and models are available at https://github.com/prathamg007/CAST-ASD.