Reinforcement learning improves crystal design with higher yield and warns of gaming risks
Reinforcement Learning on the Discrete Composition Channel of a Crystal Generator: Validated Gains and Reward Hacking
Machine Learning
Summary
Designing new materials by predicting crystal structures is challenging because existing models focus on matching known examples rather than specific goals. The authors used a type of reinforcement learning to teach a model to generate crystals with better qualities, increasing the number of useful and novel structures found. However, they discovered that the model could exploit the scoring system in unintended ways, so safeguards are necessary. They also highlight that current benchmarks may overestimate success by counting similar structures as new materials.
What this means in practice
- •For materials engineers: Improve computational design workflows by applying reinforced generative models that yield more unique and stable crystal structures.
- •For chemical product developers: Use enhanced crystal structure generation methods to explore novel compounds potentially useful for new materials or chemicals.
Authors
Pawan Prakash, Philipp Höllmer, Addis Fuhr, Peter Hirschfeld, P. Ganesh, Stefano Martiniani, Richard Hennig
Abstract
Inverse materials design is a long-standing goal of computational materials discovery. Generative models for crystalline materials are typically trained to match the distribution of a structure database, while nothing in their training objective points them at specific design goals such as targeted properties. We use group-relative policy optimization (GRPO) to align a generative model based on stochastic interpolants and discrete flow matching with general black-box reward functions through reinforcement learning. Atom types are generated by a discrete flow and the policy gradient of our generalization of GRPO directly acts on the likelihoods of the atom-type transitions, which differentiates our work from previous reinforcement-learning approaches for diffusion and flow-based generative models of crystalline materials. We introduce a reward function that raises the yield of metastable, unique and novel structures (mSUN) from 13.4% for the pretrained model to 45.5% for the reinforced model, as evaluated by a community benchmark. Our reward also improves the performance of a reinforcement learning framework for crystalline materials based on latent denoising diffusion models. At the same time, we find that directly reinforcing atom-type transition likelihoods enables reward exploitation that has to be prevented with explicit guards. The same analysis also exposes a gap in the community metric. Single-element structures in distinct packings are counted as metastable, unique and novel materials and inflate mSUN without yielding any new compounds. A stability claim is only as good as its reference hull. We report every result split by the number of reference phases behind it and argue that benchmarks should do the same.