Quantum transformer models generate synthetic languages with grammar rules
Variational Quantum Transformer Architecture for Synthetic Language Generation
Computation and LanguageMachine Learning
Summary
This paper studies how quantum computers can be used to create language models similar to those used for understanding and generating text. The authors designed a small quantum version of the popular transformer model, replacing some parts with quantum circuits that process information differently. Their quantum models learned to produce simple languages with correct grammar, although classical models still performed better. This work shows a practical way to build quantum language models that run on near-term quantum devices, without claiming they outperform classical methods yet.
What this means in practice
- •For quantum computing engineers: Develop quantum language models on near-term devices using transformer-like architectures adapted for quantum circuits.
- •For natural language processing teams: Experiment with alternative architectures for sequence modeling that integrate quantum circuit processing for grammar learning tasks.
Authors
Julian Hager, Michael Kölle, Gerhard Stenzel, Tobias Rohe, Jonas Stein, Claudia Linnhoff-Popien
Abstract
We propose a compact NISQ-compatible quantum transformer architecture for synthetic QNLP sequence modelling. The model preserves the autoregressive next-token interface of a classical transformer, but replaces attention and feed-forward sublayers with variational quantum encoder blocks, connector circuits, decoder blocks and a direct two-qubit measurement readout. Token contexts are angle-encoded into small quantum registers, processed by parallel variational heads and encoder integration circuits and conditioned through decoder ancillae to produce a distribution over a four-token vocabulary. We evaluate several architecture variants on deterministic and lexicographic grammar-generation tasks against a compact classical transformer baseline. The quantum models are trainable end-to-end and learn nontrivial grammar structure, including perfect deterministic generation in individual runs and high lexicographic validity in the strongest variant. The classical baseline remains more accurate and stable and the quantum models are sensitive to initialization. The contribution is therefore not a claim of quantum advantage, but a concrete architecture and evaluation of transformer-inspired QNLP sequence modelling under near-term quantum constraints.