AI summaryⓘ
The authors created Turing-20B-A2B, a large language model that uses a smart method to pick only about 2 billion important parts to process each word, making it efficient for tasks needing long memory and quick responses. They designed a special routing system to balance work among different experts and keep computing costs steady. The model mixes fast attention techniques with a few full attention layers to handle very long texts effectively and was trained in stages to handle really long contexts, up to 512,000 tokens. Even with fewer active parts working at once, it matches or beats other similar-sized models in overall ability and speed. Their work shows a good mix of power, long-text memory, and practical speed.
Mixture-of-ExpertsLanguage ModelQuantile RoutingTop-k ConfigurationCapacity-Constrained RoutingLightning AttentionLong-Context ModelingCurriculum PretrainingExtended Context LengthInference Efficiency
Authors
Yuheng Zhang, Yizhao Wang, Da Zhu, Hua Zhou, Yue He, Jiahui Hu, Shaman Tang, Hanlin Chen, Yuhua Wei, Anhua Liu, Shuang Su, Rui Xin, MingYuan Wang, MingHao Li, HaoJie Yang, Siqi Liu, Jianlei Zheng, WeiChao Huang, Qiman Wu, Hang Zhang, HongGou Yang, Xianming Liu
Abstract
We present Turing-20B-A2B, a 20B-parameter Mixture-of-Experts language model that activates approximately 2B parameters per token, designed for long-context and latency-sensitive physical AI applications. The model adopts Quantile Routing in a dynamic top-k configuration, enabling token-adaptive expert allocation while maintaining balanced expert utilization and a controlled average compute budget. During deployment, we further apply capacity-constrained routing to prompt prefill for more regular and efficient expert execution, while retaining dropless routing during pretraining. Turing-20B-A2B also employs a hybrid attention architecture that combines Lightning Attention with a small number of full-attention layers for efficient long-context modeling. The model is pretrained with a progressive three-stage curriculum and extended to a native context length of 128K through continued pretraining, with further inference-time extension to 512K using YaRN. Despite its compact active-parameter budget, Turing-20B-A2B achieves, at the base-model stage, overall general capability exceeding Qwen3-8B Base and approaching Qwen3.5-9B Base, while maintaining strong long-context performance and favorable prefill-latency scaling. These results demonstrate an effective balance among model capability, long-context scalability, and practical inference efficiency.