Public platform tracks and predicts large language model designs
ArchSpace: Recording, Reviewing, and Referencing LLM Architecture Research
Computation and LanguageMachine Learning
Summary
Design of language models affects how well they learn and how fast they work, but details on what works are often missing in reports. The authors created ArchSpace, a public tool where people can log design ideas, share tests, and discuss what works and why. It uses math from smaller experiments to guess how models will perform when much bigger. This helps save time and resources by showing which design ideas might succeed before expensive tests. ArchSpace also lets successful designs become new standards that others can compare against.
What this means in practice
- •For machine learning engineers: Access a public platform to record and review model architecture innovations and predict their impact at scale before costly training.
- •For cloud service operators: Use predictive scaling laws from smaller tests to better estimate resource needs and costs when deploying large language models.
Authors
Jie Wang, Tong Zhu, JiaHao Kuang, Zhe Xu, Yunkai Chai, Dehong Zheng, Yu Cheng, Tao Ji, ZhouHan Lin, Qi Zhang, Bowen Zhou
Abstract
Model architecture shapes the scaling behavior, training efficiency, and inference cost of LLMs, making it a central focus of both industry and academia. In industry, teams such as those behind DeepSeek, Qwen, and Kimi routinely introduce architectural innovations, yet public reports rarely document which ideas succeeded, which were abandoned, and why. In academia, limited compute and inconsistent baselines often restrict evaluations to small-scale, single-point comparisons, making it difficult to establish whether improvements hold at larger scales or translate into broader impact. ArchSpace addresses both gaps as a public platform for Recording, Reviewing, and Referencing Architecture Researc}: each proposal is recorded with its experiments and reasoning, reviewed by the community, and referenced by later work. To evaluate ideas before costly large-scale training, ArchSpace leverages scaling laws to predict model loss at larger scales from smaller experiments, a practice widely adopted in industry. We provide initial references that make the platform usable: Qwen3.5 and OLMo3 baselines, fitted scaling laws, and a reusable training recipe. Ideas validated through this review can in turn become new baselines, and ArchSpace thereby aims to broaden the range of ideas that shape the next generation of models.