Audio visual segmentation runs faster and simpler with encoder only model
Less is More: Encoder-only Audio-Visual Segmentation
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Audio-Visual Semantic Segmentation is a process that finds and highlights objects making sounds in videos. The authors found that existing models use extra parts that don’t actually help much. They created a simpler model called EASE that is three times faster and just as accurate. This model also trains quickly and works well on different video qualities and sizes.
What this means in practice
- •For video software developers: Process videos in real time to detect and highlight sound-emitting objects faster using a lightweight and efficient model.
- •For surveillance system engineers: Improve object detection speed in audio-visual security footage to enable quicker response times without losing accuracy.
Authors
Ilpo Viertola, Vladimir Iashin, Sophie Tötterström, Esa Rahtu
Abstract
Audio-Visual Semantic Segmentation (AVSS) aims to identify, segment, and classify sound-emitting objects in video frames. Previous Transformer-based AVSS approaches largely inherit design principles from image segmentation models. Recent studies show that these image segmentation models contain redundant components that contribute little to the segmentation performance. Following this insight, we propose Encoder-only Audio-Visual Segmentation (EASE). EASE runs at up to 365 FPS, 3x faster than prior State-of-the-Art (SotA) AVS models at comparable accuracy, and trains in under 11 GPU-hours. Furthermore, we achieve SotA AVSS performance across different backbones and input resolutions. Our results demonstrate that AVSS can be both simpler and faster, providing a scalable foundation for future research and real-time applications. Code, model weights, and samples are available at https://ease-avs.notion.site