Elastic selective spectral hybrids enable flexible language model efficiency
Elastic Selective Spectral Hybrids for Train-Once, Export-Many Budgeted Inference
Machine Learning
Summary
Different uses of language models require different balances between speed and accuracy. The authors propose a new model called Elastic Selective Spectral Hybrid (ESSH) that allows the model to adjust its size and computation dynamically without losing important information. ESSH uses a special way to remember and forget input information based on context, making it more efficient. This model works well across various tasks like text and DNA processing and runs faster than some comparable alternatives.
What this means in practice
- •For ai model deployment teams: Run a single language model that can adjust quickly for different device speeds and latency constraints without retraining separate smaller models.
- •For biotech data engineers: Process DNA sequences with adaptable model quality and speed, balancing computational cost and accuracy in biological data analysis.
Authors
Dachuan Song, Chuchu Chen, Xuan Wang
Abstract
Deploying a language model under different computing and latency budgets calls for compact models with different quality-cost trade-offs. Towards this end, elastic spectral state space models provide ordered and temporally decomposed channels that can be truncated, but they are associated with linear time-invariant filters that cannot selectively preserve relevant past information or forget irrelevant information as the context evolves. To address this, we introduce the Elastic Selective Spectral Hybrid (ESSH), which realizes each Hankel spectral channel as an independent recurrent unit using fitted damped rotation modes. It also features an input-dependent decay and write/read gates that make temporal retention and state update input-dependent while preserving channel-wise truncation and a structured recurrence for efficient execution. ESSH combines these selective spectral mixers with sliding-window attention and jointly trains multiple capacities by reducing spectral-channel count and feed-forward width at different rates through a two-rate capacity map with full-model distillation. The resulting models support chunked parallel training and fused recurrent decoding while avoiding computation for discarded channels. At full capacity, ESSH achieves language-modeling quality comparable to similarly sized independently trained models, while smaller exports exhibit a smooth quality-cost trade-off. We validate the effectiveness of the proposed framework using language understanding, retrieval, cross-domain text, and DNA experiments, by assessing quality retention and the trade-off against independently trained and elastic baselines. At the 1.53B model configuration, fused batch-one decoding takes 1.37 ms per token on a B300, providing a 2.14-2.80x speedup over the tested Mamba-2 and Mamba-3 implementations and 3.03x over Transformer++ at matched parameter counts.