Layer adaptive compression improves audio codec efficiency and quality
LACE: Layer-Wise Compression for Dynamic Frame Rate Codecs
SoundArtificial IntelligenceComputation and Language
Summary
Audio codecs convert speech into digital signals but work with very fast frame rates, which can be slow and costly to process. The authors introduce LACE, which compresses audio differently at each level of processing, matching how much the sound changes over time. This method helps speed up text-to-speech systems without losing the quality of the speech. Tests show LACE is better than older methods at balancing speed and accuracy in recreating speech.
What this means in practice
- •For speech synthesis developers: Increase the speed of text-to-speech systems by using layer-adaptive compression to reduce computational demands without sacrificing speech quality.
- •For audio codec engineers: Design improved neural audio codecs that apply layer-specific compression steps to better handle variable sound changes per layer for efficient encoding.
Authors
Thanapat Trachu, Samuele Cornell, William Chen, Shinji Watanabe
Abstract
Neural audio codecs are a key component in speech language modeling. However, their high frame rates lead to long sequence lengths, increasing computational costs. Dynamic frame rate codecs mitigate this by reducing the effective frame rate using a compression step to merge multiple frames together. However, most prior methods either operate on single-codebook codecs or apply a single compression step before multi-layer quantization. This forces all quantization layers to share the same segmentation boundaries, despite the residual embeddings at different quantization layers exhibiting different rates of change over time. We propose LACE (Layer-Adaptive Codec Encoding), a dynamic frame rate codec that applies an independent compression step at each quantization layer, enabling layer-specific segmentation boundaries. To use LACE tokens in downstream text-to-speech (TTS), we further introduce union alignment and boundary anchor mechanisms to make durations consistent across layers while preserving compression benefits. Experiments on LibriTTS show that LACE offers a better rate-quality tradeoff than prior dynamic frame rate methods on the reconstruction task and improves TTS inference efficiency while maintaining competitive synthesis quality. Our code is released as part of the ESPnet3 codec recipe.