Fusion Organ network cut transformer memory use with equal accuracy

FON: A Draft-Revise Dual-Organ Language Model without Sequence-Level Quadratic Attention

Artificial Intelligence

Summary

Reading and mixing all parts of a long text in language models is usually slow because they compare every part with every other part, which takes a lot of work. The authors created a new model called FON that splits this task into two parts: one quickly drafts text based on recent information without heavy calculations, and another revises the draft by reviewing what’s written so far without producing new text directly. This approach uses less memory and runs faster while keeping accuracy better than many earlier models, especially on very long texts. However, it does have some weaknesses like slower real-time speed and less effective handling of certain code-related tasks compared to traditional models.

What this means in practice

Authors

Helong Hu, Hongdan Pan, Shuiqing Hu

Abstract

Self-attention assigns two functions to one mechanism-where to read and how to mix-at the price of O(L^2) training, an O(T) decode state, and position signals bound to the training length. Existing attention-free alternatives all optimize the single function of generation. We propose FON (Fusion-Organ Network), which differentiates one weight stack into a drafting organ (multi-tap learnable delays + content-adaptive recursion; causal, O(S) training, O(1) decode state, no sequence-level quadratic attention) and a revision organ that re-processes the written prefix through multi-scale ripple kernels injected via weak gates, never generating. At 30M parameters, iso-parameter and iso-token budget, FON reaches perplexity 131.03 on WikiText-103, beating Transformer (197.73) and Mamba2 (152.53); the advantage persists under all depth-aligned baselines and at the 70M tier (-20.8%). At 8x length extrapolation FON degrades by ~8-9% without positional-encoding engineering, between ALiBi and RoPE. The revision channel provides a structural pathway for in-domain infilling that causal architectures lack in principle (+19% over on-demand coupling), at a perplexity cost indistinguishable within measurement precision. We report all costs and boundaries: impaired long-range retrieval, synthetic-task disadvantages, weaker out-of-domain code infilling than a same-recipe Transformer, and a 6.5-12.3x wall-clock gap.
View PDFOpen arXiv