Length based training shows no clear benefit for speech models
Rethinking Length-Based Training: Batch Composition and Loss Normalization in Speech Token Language Models
Computation and Language
Summary
Training speech models by starting with short audio clips and moving to longer ones is a common technique. The authors show that after carefully matching batch details and token exposure, this short-to-long training doesn't independently improve the model's understanding. They also find that how batches are grouped and how different tokens are weighted during training can affect results. Their work provides a clear way to study training methods for speech models that handle varying lengths.
What this means in practice
- •For speech recognition engineers: Design training schedules for speech models with variable input length considering batch composition to optimize performance.
- •For machine learning platform teams: Implement systematic analysis protocols to better evaluate length-based training strategies in speech model pipelines.
Authors
Hongjin Song, Runwu Shi, Weiqiao Shan, Jiale Luo, Yujin Wang, Yifei Wu, Chunxiang Jin
Abstract
Short-to-long training is a simple curriculum for speech models, but its gains can be difficult to interpret. In speech token language models, length-based training can change the shuffle policy, batch composition, token retention, and token weights under batch-mean loss. We disentangle these factors through matched comparisons. In the tested settings, short-to-long ordering shows no independent benefit when batch composition and token exposure are fixed. First-epoch grouping lowers perplexity for Mimi under batch-mean loss, but this gain is not observed under token-balanced loss. The cross-tokenizer results are consistent with a link between chunk-length variation and token weighting. This work provides a systematic analysis protocol for studying length-based training in variable-length speech models.