Graph transformers perform better when trained on computed biochemistry properties
Pre-training with Graph Transformers
Machine Learning
Summary
Understanding complex molecules can help us develop new medicines and materials. The authors study a type of AI model called graph transformers that look at molecules as connected points. They found that teaching these models first by showing them computed chemical properties helps them learn better and perform well on later tasks. Also, keeping the model size smaller helps avoid learning mistakes from too much data.
What this means in practice
- •For pharmaceutical developers: Improve molecule property prediction models by using supervised pre-training on computed biochemistry labels to aid drug design workflows.
- •For chemical simulation teams: Build better molecular models by controlling model complexity to reduce overfitting when using graph transformer architectures.
Authors
Jiaming Wang, Thomas Laurent, Xavier Bresson
Abstract
This article investigates pre-training strategies for graph transformers in the biochemistry domain. By conducting comprehensive experiments, the study reveals that supervised pre-training using computed properties as labels provides the highest performance gain on downstream tasks. The results also highlight the importance of constraining model capacity to mitigate overfitting in graph transformers.