Summary
Training large AI language models is often not fully reproducible because small differences in hardware and software can cause different results, even when using the same data and code. The authors introduce a new way to train models so every training step can be exactly repeated on any standard computer, making the entire training process auditable. They do this by carefully controlling sources of randomness and ordering operations so that even complex distributed training can be reliably checked. To make auditing scalable, many independent auditors verify different training steps rather than one machine replaying everything. They demonstrate this with Open-1B, a model released along with its full training data, checkpoints, code, and tools for anyone to verify its training history.
What this means in practice
- •For machine learning engineers: Verify that a released language model checkpoint was genuinely produced by the declared training steps on diverse hardware with exact reproducibility guarantees.
- •For cloud infrastructure teams: Support scalable, distributed model training audits where different training steps can be independently checked for correctness across multiple machines.
Authors
John Donaghy, Brian Wilcox, Oğuzhan Ersoy, Shikhar Rastogi, Adam St Arnaud, Alexey Titov, Jordan Greenberg, Ben Fielding, Harry Grieve
Abstract
Open-source language models have a reproducibility problem. Despite releasing weights, training data, and recipes, none of them are provably reproducible due to the non-associativity of floating-point arithmetic. Deep learning frameworks often offer a deterministic execution mode, allowing reproducible operations on the same machines. Unfortunately, this determinism does not carry across hardware such that a user can verify that a released checkpoint was actually produced using the declared training recipe. This leaves room for undisclosed data, injected biases, or backdoors that existing techniques such as proof-of-learning or proof-of-training-data cannot rule out. We introduce a new tier of model transparency, fully auditable, in which every operation on every data sample during training is independently reproducible on heterogeneous commodity hardware with bitwise certainty. By imposing a definite order on the sources of training nondeterminism, GPU kernel reductions, data batch ordering across a data-parallel cluster, and inter/intra-node collective communication, we make it possible to replay any individual step of a large, distributed training run on a single piece of commodity hardware and check it against the published trajectory. Because replaying an entire run on one machine is infeasible, we support this with a collective verification scheme in which many independent auditors each certify individual steps, together covering the whole run. We release Open-1B, a model trained under this regime, together with its full pretraining dataset, every intermediate checkpoint, the training codebase, and the audit harness needed to reproduce and verify any step of its training.