Probing Chemical Language Models: Effects of Pre-training and Fine-tuning
2026-07-02 • Machine Learning
Machine Learning
AI summaryⓘ
The authors studied how chemical language models (CLMs), which read molecules as strings, recognize different parts of molecules. They tested 78 common molecular features on both trained and untrained models and found that training helps the models understand molecular structures better, especially in deeper layers. Interestingly, even untrained models noticed ring structures early on. When the models were fine-tuned for specific chemistry tasks, they adjusted their focus on molecular parts important for those tasks, matching chemical knowledge.
Chemical Language ModelsSMILESMolecular SubstructuresPre-trainingFine-tuningRepresentation LearningRing StructuresDownstream TasksMolecular FeaturesChemical Informatics
Authors
Anna Karnysheva, Dietrich Klakow, Ji-Ung Lee
Abstract
Chemical language models (CLMs) are trained with linearized representations such as SMILES, yet it remains unclear which chemically meaningful substructures they encode. To foster a better understanding of CLMs, we conduct a systematic study and probe for 78 molecular substructures across eight pre-trained and six randomly initialized models. We furthermore study how fine-tuning on chemical downstream tasks affects the learned representations of molecular substructures. Our results show that pre-training generally improves molecular structure awareness of CLMs, particularly in the upper layers. Moreover, randomly initialized models already encode ring structures well in the first layer. Our analysis on two chemical downstream tasks further reveals that, interestingly, fine-tuning affects task-relevant molecular substructures more than others, indicating that the changes in the representations follow chemical theory.