Orthros achieves faster transformer attention with phase aware switching
Orthros: Phase-Aware Heterogeneous Attention for Efficient Transformers
Machine LearningArtificial Intelligence
Summary
Transformers are a type of AI model that can read and understand long texts, but their attention process usually gets slower as the text gets longer. The authors present Orthros attention, a method that smartly switches how it processes parts of the text in phases, making it much faster especially when reading a lot at once and then focusing on smaller parts. Tests show Orthros keeps accuracy while running up to 22 times faster during the big reading step, making it promising for real-world uses with very long texts.
What this means in practice
- •For ai system engineers: Speed up AI models handling long documents by implementing Orthros attention to efficiently process prefilling and decoding phases.
- •For cloud ai platform teams: Reduce inference latency and resource use for large-scale transformer deployments with Orthros, enabling cost-effective long-context AI services.$Commercial implications: Enables cloud providers to offer faster and more efficient transformer inference for customers needing long-form text processing.
Authors
Yuhong Chou, Zehao Liu, Yuqi Pan, Qian Liu, Xianwei Chen, Zhongzhu Zhou, Jibin Wu
Abstract
Efficient attention mechanisms, such as linear and sliding window attention, aim to reduce the computational complexity of Transformers from quadratic to linear. However, their degraded in-context recall often necessitates interleaving full attention layers, leaving the quadratic bottleneck unresolved in long-context processing. To address this, we introduce Orthros attention, which shifts the efficiency paradigm from structural allocation to phase-aware decoupling. As a unified module, it dynamically switches its computation pattern across different phases of sequence processing, enabling linear complexity prefilling and full attention decoding. Our design grants Orthros attention a lower asymptotic complexity compared to vanilla Transformers in prevalent long-context scenarios characterized by heavy prefilling and light decoding. Beyond theoretical formulation and complexity analysis, extensive empirical evaluations demonstrate Orthros's superiority across multiple dimensions. First, we validate its fundamental language modeling capabilities, demonstrating scalability across varying model sizes and strong competitiveness in comprehensive architectural comparisons. Second, Orthros exhibits exceptional conversion feasibility via a data-efficient Transformer-to-Orthros conversion, yielding performance that even surpasses the original Transformer. Finally, Orthros achieves a $21.9\times$ prefilling speedup over Transformers with end-to-end inference gains, while preserving long-context capabilities, as demonstrated by near-perfect 128K needle-in-a-haystack results and Transformer-level performance on real-world in-context recall tasks. Overall, these advantages in high performance, flexibility, and efficiency position Orthros as a promising candidate to succeed the Transformer in industrial applications.