Small recursive models improve interpretability and match gpt 2 performance
Stream Recursion Model (SRM)
Machine Learning
Summary
Understanding how large AI language models work inside is hard because they are very big and complex. This paper introduces a new smaller model called Stream Recursion Model (SRM) that splits its thinking into several parts that talk to each other and improve their understanding step-by-step. This structure makes it easier to analyze exactly how the model processes information and decides what to do. The researchers showed that SRM performs about as well as GPT-2 when comparing similar sizes of models. They also found that each part of SRM behaves in distinct and consistent ways, suggesting the model organizes its work clearly.
What this means in practice
- •For machine learning engineers: Design smaller language models with clearer internal structures to debug and improve reasoning more effectively.
- •For ai safety teams: Analyze internal decision-making in language models by tracing how different information streams interact and influence outputs.
Authors
Asael Sorensen, Charles Brock, David Chamberlain, Jennifer Minnich, Matthew Hoffman, Ramyaa Ramyaa
Abstract
Mechanistic interpretability seeks to make verifiable statements about the internal behavior of large language models (LLMs). Many interpretability techniques struggle to scale with the increasing size and depth of architectures. Our solution to this is to introduce smaller models with structures that lend themselves to interpretability. In this work, we introduce the Stream Recursion Model (SRM), a modification of the Hierarchical Reasoning Model (HRM) designed to expose internal computational structure while remaining scalable. SRM organizes computation into multiple interacting latent streams that are updated through recursive refinement, enabling direct analysis of stream dynamics, causal contribution, and routing behavior. SRM achieves performance comparable to GPT-2 on a per-parameter basis. Our analysis reveals consistent and distinct behavior across streams, indicating structured specialization and interaction. These results suggest that SRM provides a practical architectural foundation for scalable mechanistic interpretability and opens up promising avenues for future research in both reasoning performance and interpretability.