Contrastive audio language models improve with larger diverse datasets

LAION-Audio-XL: A Web-Scale Corpus and Scaling Studies for Contrastive Language-Audio Pretraining

SoundComputation and Language

Summary

Understanding sounds and matching them with text descriptions is a tricky problem. The authors gathered over a billion diverse audio clips from many sources and cleaned them carefully. They added captions when missing and trained models that connect sounds and words better than before. By studying how bigger models and more data affect results, they found good trade-offs for performance. Their trained models beat previous ones on popular benchmarks and are shared for others to use.

What this means in practice

  • For audio software developers: Build applications that recognize and describe a wide range of sounds by using the large curated audio-caption dataset and pretrained models.
  • For multimedia content platforms: Enhance search and content recommendation by integrating improved audio-language matching models trained on diverse web-scale datasets.

Authors

Gijs Wijngaard, Felix Förster, Ross Wightman, Marianna Nezhurina, Christoph Schuhmann, Jenia Jitsev, A. Sophia Koepke

Abstract

Recent progress on contrastive audio-language models can be attributed mainly to the scale and quality of their training data. Larger training datasets expose models to a broader range of sounds and corresponding audio captions. To exploit this, we collect 1.48 billion audio samples from academic datasets, internet datasets, web collections, Common Crawl, and YouTube, and process them with a multi-stage curation pipeline. Furthermore, we generate short and long captions for samples in our corpus that lack them. We use our dataset to study the scaling behavior of contrastive audio-language models based on the NaFlex architecture for the audio encoder. Our experiments identify trade-offs between model size and training data for a given compute budget. Moreover, our resulting models, CLAP2, outperform prior audio-text models of comparable size on the MAEB and HEAR benchmarks. We release our dataset, captions, model checkpoints, and code.