Reinforcement Learning with LLM-Guided Action Spaces for Synthesizable Lead Optimization
2026-04-09 • Machine Learning
Machine LearningArtificial IntelligenceComputational Engineering, Finance, and Science
AI summaryⓘ
The authors designed MolReAct, a method to help improve drug molecules while making sure the changes can actually be made in a lab. Their approach uses a smart program that thinks step-by-step about how to safely change a molecule based on real chemical reactions. By combining this with specialized tools and a learning system, they can quickly find better drug candidates and explain how to make them. Their tests show MolReAct does better than other methods, making molecules that are both improved and practical to synthesize.
lead optimizationdrug discoveryMarkov Decision Processreaction templatesLarge Language Modelschemical synthesispolicy optimizationSMILESTherapeutic Data Commonsdocking task
Authors
Tao Li, Kaiyuan Hou, Tuan Vinh, Monika Raj, Zhichun Guo, Carl Yang
Abstract
Lead optimization in drug discovery requires improving therapeutic properties while ensuring that proposed molecular modifications correspond to feasible synthetic routes. Existing approaches either prioritize property scores without enforcing synthesizability, or rely on expensive enumeration over large reaction networks, while direct application of Large Language Models (LLMs) frequently produces chemically invalid structures. We introduce MolReAct, a framework that formulates lead optimization as a Markov Decision Process over a synthesis-constrained action space defined by validated reaction templates. A tool-augmented LLM agent serves as a dynamic reaction environment that invokes specialized chemical analysis tools to identify reactive sites and propose chemically grounded transformations from matched templates. A policy model trained via Group Relative Policy Optimization (GRPO) selects among these constrained actions to maximize long-term oracle reward across multi-step reaction trajectories. A SMILES-based caching mechanism further reduces end-to-end optimization time by approximately 43%. Across 13 property optimization tasks from the Therapeutic Data Commons and one structure-based docking task, MolReAct achieves an average Top-10 score of 0.563, outperforming the strongest synthesizable baseline by 10.4% in relative improvement, and attains the best sample efficiency on 10 of 14 tasks. Ablations confirm that both tool-augmented reaction proposals and trajectory-level policy optimization contribute complementary gains. By grounding every step in validated reaction templates, MolReAct produces molecules that are property-improved and each accompanied by an explicit synthetic pathway.