What Does a Chemical Language Model Know About Molecules?
2026-06-22 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors studied how a chemical language model called MolFormer understands molecules by looking inside its layers using a technique called sparse autoencoders. They found that early layers mostly keep track of the position of atoms to understand the molecule's grammar, while later layers focus on meaningful features like atom environments and drug-related properties. They also discovered that unusual but valid molecule representations cause more changes inside the model than invalid ones. To help others explore these findings, the authors created a tool called InterMol that shows how the model processes molecules step-by-step.
chemical language modelsMolFormersparse autoencodersmolecular representationsSMILES notationencoder layersatom-in-substructurepharmacological featuresnon-canonical SMILESmodel visualization
Authors
Christian Kenneth, Etowah Adams, Liam Bai, Gerard JP van Westen
Abstract
Chemical language models (cLMs) are widely assumed to learn surface-level syntactic patterns rather than learning meaningful molecular semantics. Here, we apply sparse autoencoders (SAEs) to MolFormer, an encoder-only cLM, to mechanistically examine how molecular representations are built across layers. We discover that early layers rely on position-tracking latents to parse molecular grammar, while later layers encode atom-in-substructure and pharmacologically relevant features. Additionally, we show that non-canonical SMILES produce more disruptive representation shifts than invalid SMILES, driven by position-latent disruption propagating across layers. To support further exploration, we develop InterMol, an interactive visualizer for SAE activations on molecular strings and structures.