Speedrun leaderboard speeds up relational transformer training times

Speedrunning Relational Foundation Model Pretraining

Artificial Intelligence

Summary

Training large models that understand relationships between data points can be very expensive and private, limiting progress. The authors introduce a public leaderboard called modded-nanoRT that ranks attempts by how fast they reach a certain quality using only one GPU. They develop a new method using 'typed trilinear attention' which trains relational transformers much faster than previous versions. This approach makes it easier for anyone to try improving relational model training speed and share results openly.

What this means in practice

  • For machine learning engineers: Fast-track relational model training experiments by using the modded-nanoRT leaderboard and recipe to reduce iteration times significantly on single GPUs.
  • For database system developers: Benchmark and improve relational attention mechanisms in models handling structured or relational data with fast, reproducible training runs.

Authors

Gabriel Wedson Mendonça de Souza Veloso, Luan Borges Teodoro Reis Sena, Denis Oliveira Correa, Francisco Galuppo Azevedo

Abstract

Progress on foundation model pretraining is gated by who can afford a trial, and relational pretraining is no exception. Recipes are developed inside individual labs, reported under private protocols, and do not accumulate. Speedrun leaderboards remove that barrier by decentralizing the search rather than the training run. The problem is fixed, the stack is open, entries are ranked by wall-clock time to a quality target, and one GPU is enough to hold the record. We introduce modded-nanoRT, the first such leaderboard for Relational Transformers. Which database to hold out decides whether the benchmark can resolve improvements at all, so we fix it by calibration, ranking six candidates by how sharply a 45-minute learning curve separates improvement from seed noise. We open the leaderboard with typed trilinear attention, a relational operator that conditions attention scores and values on the foreign key traversed, together with Muon and SIGReg. The recipe reaches the target in 305.63 seconds on one NVIDIA H200, 8.11x faster than the reference implementation, and wins 13 of 15 pairs outside the development database. We will release the benchmark, its frozen artifacts, and the record runs. Our code is available at https://github.com/kunumi/nanoRT.