ZetaGPT: A Reference Implementation of Positional--Encoding--Free State--Space--Attention Language Models
2026-08-10 • Computation and Language
Computation and LanguageArtificial Intelligence
AI summaryⓘ
The authors explore a new way for language models to understand word order without using usual position markers. They use something called causal state-space equations to help the model remember the order of words before the attention part of the model looks at them. This lets the model work with word positions naturally, rather than needing extra position information. They introduce ZetaGPT, a small, fully open-source language model that uses this idea and includes tools for training and fine-tuning. Their work provides a new example of a model that learns order without explicit position signals.
transformerself-attentionpositional encodingrotary positional encoding (RoPE)state-space modelscausal state-space equationslanguage modelreinforcement learning from human feedback (RLHF)chain-of-thought reasoningtokenizer
Authors
Róisín Luo
Abstract
Transformer-based language models rely on self-attention, whose computation is permutation-equivariant and therefore lacks an intrinsic mechanism for representing token order. Existing architectures address this limitation by explicitly incorporating positional information through learned positional embeddings or hand-crafted positional encodings, such as rotary positional encoding (RoPE), treating positional information as an architecturally acquired capability rather than an inherent property of the model. Motivated by the pursuit of positional-encoding-free architectures, this work explores a language model architecture that integrates causal state-space equations to implicitly encode positional information before attention computation. Specifically, each model block applies a causal state-space equation before self-attention, allowing recurrent state dynamics to encode sequential information into token representations. Consequently, subsequent attention layers operate on position-aware representations without requiring explicit positional encodings while retaining the expressive modeling capacity of self-attention. We present \textsc{ZetaGPT}, a compact hybrid language model designed for research, rapid prototyping, algorithm verification, and educational applications. In addition to the proposed architecture, \textsc{ZetaGPT} provides a fully open-source, end-to-end training pipeline encompassing dataset construction, tokenizer training, pretraining, supervised fine-tuning, reinforcement learning from human feedback (RLHF), and chain-of-thought (CoT) reasoning via pure reinforcement learning. To the best of our knowledge, \textsc{ZetaGPT} is the first open-source small language model without explicit positional encoding and establishes a compact, reproducible reference implementation for the development and empirical study of positional-encoding-free language models.