Stable transformers improve deep graph generation performance
Stable Transformers for Graph Generation
Machine Learning
Summary
Graph Transformers are tools used to create complex graph structures, but when made deeper, they can lose effectiveness as their internal signals shrink and become harder to train. The authors studied this shrinking problem and found that it causes loss of information and weakens learning. They created a new transformer design that avoids this shrinking, preserving information flow and improving how well graphs are generated, especially in deeper models. Their experiments showed that this stability leads to better results on synthetic and molecular graph tasks.
What this means in practice
- •For drug discovery teams: Generate more reliable molecular graphs for potential new drugs by using stable transformers that maintain information flow in deep models.$Commercial implications: Enables commercial molecular design software to produce better candidate structures with deeper graph models, improving drug discovery pipelines.
- •For ai developers building graph-based tools: Implement stable graph transformers to maintain strong gradient signals and representation diversity in deep generative models for complex graph outputs.
Authors
Luca Miglior, Alessio Gravina, Davide Bacciu
Abstract
Graph generative models increasingly rely on Graph Transformers (GT) to capture complex dependencies among nodes and edges. While deeper architectures should provide greater expressive capacity and a broader receptive field, their effectiveness can decline with depth: repeated self-attention progressively contracts node representations, impeding information flow and gradient propagation. We analyse this phenomenon from a dynamical systems perspective, focusing on how the denoiser's spectral dynamics affect graph generation. We show that standard GT denoisers become increasingly dissipative as depth grows, leading to vanishing gradients and representation collapse. To isolate the effect of these dynamics, we construct a permutation-equivariant GT with inherently stable, non-dissipative transport. We also introduce a damping mechanism that continuously interpolates between non-dissipative and increasingly contractive regimes, enabling a direct assessment of how dissipation influences generation. Experiments on synthetic and molecular graph generation benchmarks show that the gap between these regimes widens with depth: non-dissipative dynamics preserve representation diversity and gradient flow, sustaining strong generative performance, whereas greater contraction progressively impairs it. These findings identify the denoiser's dynamical regime as a key design factor for deep graph generative models.