Transformer model replicates causal discovery update steps accurately

Executing Causal Structure Learning with Linear-Attention Transformers

Machine Learning

Summary

Finding relationships that cause effects is hard, especially when computers need to figure it out step by step. The authors show that a special kind of AI model called a transformer can perfectly mimic one step of a known method for discovering cause-and-effect links in data. This approach repeats the steps by passing information forward multiple times, holding extra details needed for accuracy. They find normal transformer training doesn’t learn this behavior well, leaving open whether such models can be trained to do this on their own.

What this means in practice

  • For machine learning engineers: Use transformers to exactly replicate iterative causal discovery updates for debugging or extending causal analysis workflows.
  • For software developers in ai tools: Build systems that precisely execute known causal discovery algorithms within transformer architectures for integration into AI frameworks.

Authors

Amartya Roy, Sayar Karmakar

Abstract

Transformers can execute algorithms on data given in their input. We ask whether they can do the same for causal discovery. We study a standard continuous method that repeatedly updates a candidate causal graph while enforcing acyclicity. We explicitly construct a fixed-weight transformer whose forward pass exactly reproduces one update of this method, so repeated blocks reproduce its optimization trajectory. The transformer carries the current graph and the algorithm's multiplier between updates. We show that retaining the multiplier is essential for exact execution, since different multiplier values can lead to different next updates. We also give conditions under which, within a fixed stage, the number of updates needed to reach a target accuracy can be computed in advance and rounding errors stay bounded as depth grows. Experiments show that the constructed block agrees with a reference update to floating-point precision, while arithmetic replay on synthetic data and seven published benchmark network topologies inherits the reference solver's successes and failures. This separates accurate algorithm execution from accurate causal recovery. In contrast, the ordinary attention models tested under our training budgets do not reliably execute the update or transfer to larger graphs. Whether gradient training can learn an executor in the architecture class of the construction remains open.