Asymmetric looped transformers improve language model efficiency and accuracy
Asymmetric Looped Transformers: Fixed Global Representations for an Iterative Decoder
Computation and Language
Summary
Transformers are models that process language by passing information through layers repeatedly. The authors found that mixing how context is understood with how the model updates its thinking in one step can reduce effectiveness. They designed a method where the model first builds a fixed understanding of context in one pass and then refines its thoughts repeatedly. Their approach led to better performance on language tasks and reasoning tests compared to previous looping methods.
What this means in practice
- •For natural language processing engineers: Build more efficient language models that use fixed global context to improve iterative reasoning without increasing parameters.
- •For machine learning platform teams: Deploy optimized Transformer architectures that maintain accuracy while scaling training compute efficiently through asymmetric looping.
Authors
Cheng Liang, Tengfei Zhang, Wenjie DU, Jianyu Wu, Changlian Ma, Shuchen Pu, Ya Zhang, Yanfeng Wang, Zhouhan Lin
Abstract
Looped Transformers increase effective depth by repeatedly applying shared layers, offering a way to scale computation without increasing parameter count. However, full-stack looping places context construction and iterative state refinement within the same recurrent computation, without explicitly distinguishing their roles. We introduce the Asymmetric Looped Transformer (AsymLoop), which separates single-pass context encoding from recurrent state refinement. A single-pass causal encoder produces layer-specific Global key-value (KV) caches that remain fixed across iterations, while a weight-tied recurrent decoder generates Local KV from its evolving state. Joint attention over both sources under a single softmax allows each update to integrate persistent contextual information with the current intermediate state. For training, we apply a heavy-tail-guided layerwise learning-rate method to the encoder and decoder. Experiments on FineWeb-Edu across four model scales, from 60M to 1B, show consistently lower validation perplexity and higher average downstream accuracy than Full Loop under matched effective depth and training token budgets. At 1B, AsymLoop improves average zero-shot accuracy across seven commonsense reasoning benchmarks from 49.58% to 50.85%. Further experiments extend these benefits to longer training budgets, an OLMo-style backbone, and sparse mixture-of-experts models. These results support asymmetric context encoding and state refinement as an effective design for recurrent language models.