Models learn relations when data forces cross-table predictions

When Does Synthetic Relational Data Teach Models to Use Relations? Tracing Predictive Structure from Pretraining Data to Model Behavior

Machine Learning

Summary

The paper looks at how artificial databases help AI models learn to use relationships between tables. The researchers found that models only develop the ability to link information across tables if the training data actually requires this to succeed. They traced this effect by comparing models trained on different types of synthetic data and showed that certain data properties make models rely on relational connections. When those connections are broken during testing, only the models trained on the right data lose performance on relational tasks. This shows which data properties teach models to handle relations.

What this means in practice

  • For machine learning engineers: Design synthetic training datasets that encourage models to use relational structure for better performance on relational tasks.
  • For database system developers: Create benchmark data generators that reveal whether models truly learn relational reasoning rather than exploiting surface statistics.

Authors

Shivam Dubey, Mohamed Bouadi, Nassim Bouarour, Varun Kulkarni, Aditya Tanna, Vinay Kumar Sankarapu

Abstract

Relational foundation models are increasingly pretrained on synthetic databases, yet downstream benchmarks reveal little about why one synthetic corpus produces a better model than another. In particular, strong performance may arise from realistic row-level statistics without the model ever learning to use relational structure. We study this as a data-attribution problem: which property of synthetic pretraining data induces relational computation? Using four Relational Transformer checkpoints trained with the same architecture, initialization, objective, and compute budget on corpora produced by four relational data generators, we trace a measurable property of the data to learned computation and downstream behavior. We hypothesize that relational mechanisms emerge when cross-table information is predictively necessary for the masked-cell pretraining objective. RelDiff exhibits by far the largest predictive gain from foreign-key-linked parents, and its corresponding model is uniquely sensitive to foreign-key interventions on unseen databases. This dependence survives a random-initialization control, grows monotonically with the fraction of corrupted links, and localizes to a serial cross-table pathway. Finally, disrupting the same mechanism during downstream inference removes RelDiff's advantage on relational tasks while leaving structure-insensitive models nearly unchanged. These results connect a property of synthetic training data to a learned mechanism and, through intervention, to downstream behavior.