Quality gated layers convert pretrained attention in language models
TinyCeNN-LM: Quality-Gated Conversion of Pretrained Attention with CeNN-Inspired Cellular-Recurrent Layers
Artificial Intelligence
Summary
Language models use a technique called attention to understand text, but swapping this out after training can harm how well the model works later. The authors present TinyCeNN-LM, a careful step-by-step method that replaces attention with special recurrent layers inspired by cellular networks without losing too much accuracy. They test their method on different large language models showing it keeps performance stable by only accepting new layers when they meet strict quality checks. This helps keep models efficient while maintaining the expected behavior.
What this means in practice
- •For large language model engineers: Replace parts of pretrained attention layers with efficient recurrent layers without harming model accuracy by using quality-gated conversion layers.
- •For natural language processing platform teams: Implement progressive layer substitution in language models to reduce memory caches while monitoring representation fidelity and prediction quality.
Authors
Kabeh Mohsenzadegan, Vahid Tavakkoli, Kyandoghere Kyamakya
Abstract
Replacing attention in a pretrained language model is a compatibility problem: a plausible substitute may alter representations expected by later layers. TinyCeNN-LM introduces a \emph{quality-gated post-training conversion} framework using CeNN-inspired cellular-recurrent layers with bounded local processing, compact recurrent memory, routing, fusion, and accept-or-rollback validation. Three implementations are studied: Integrated Memory, MemoryFusion, and PDelta3-GDN2-CLVR+Local32. Strict PDelta3 conversion accepts a layer only when representation and NLL criteria pass fixed thresholds. On SmolLM2-135M, layers 0-2 are accepted with cumulative $Δ\mathrm{NLL}=+0.01209$, while layer 3 is rejected despite acceptable NLL because representation fidelity fails. On Qwen3.5-0.8B, full-attention layers 3, 7, and 11 are accepted with final $Δ\mathrm{NLL}=+0.02073$. Integrated Memory keeps perplexity within $-0.07\%$ to $+0.93\%$ while reducing total cache by up to $6.01\%$. A sampled 200-item downstream sanity check gives $28.5\%$--$32.0\%$ overall accuracy for converted Qwen releases. The results support conservative, quality-gated structural conversion rather than universal attention replacement or speedup.