Temporal recurrence reduces needed network layers with similar performance
Temporal Recurrence Favors Fewer Layers
Machine Learning
Summary
Neural networks that process streaming data can remember information over time, so they don’t need to do as much work within each step. The paper studies how much layering a model needs when it already uses this kind of memory. The authors compare different setups with the same computing power and find that models using time-based memory do better with fewer layers per step. This means for some tasks like puzzles or language prediction, simpler per-step networks are enough if the model remembers past steps well.
What this means in practice
- •For machine learning engineers: Design more efficient streaming models by reducing layers per step when recurrence is used, saving computation while maintaining accuracy.
- •For natural language processing teams: Develop language models with fewer layers at each step if leveraging temporal recurrence, improving inference speed on sequential data.
Authors
Ivan Anokhin, Johan Obando-Ceron, Irina Rish, Sebastian Risi
Abstract
In streaming tasks, recurrent models can carry latent computation across time, allowing each update to build on representations produced earlier. This raises a basic question: once temporal recurrence provides sequential computation across steps, how much depth is still needed within each step? Prior work has shown that recurrence can make shallow models competitive. We instead study this question as a compute-allocation problem, varying within-step depth, expert width, and the number of parallel experts per layer across several compute budgets. For each budget, we compare the best observed recurrent and non-recurrent allocations and the performance they achieve under approximately matched per-step computation. Across Sokoban and autoregressive FineWeb language modeling, we find that temporal recurrence shifts the best observed compute allocation toward substantially fewer layers, with comparable or better performance.