Large language models can design risky biological mutations with tools
DRYLAB-Bench: Benchmarking the Biosecurity Risk of Agentic LLM Mutation Design
Artificial Intelligence
Summary
As language AI models grow better at designing biology, they can accidentally create harmful changes like stronger viruses or drug-resistant bacteria. The authors introduce DRYLAB-Bench, a way to test how well these models can design risky mutations using biological design tools and real experimental data. Their tests show many AI designs are actually quite dangerous, especially when the AI uses extra tools. This benchmark helps measure and understand these risks safely and at scale.
What this means in practice
- •For biosecurity teams: Use DRYLAB-Bench to assess and monitor the biosecurity risk of AI-designed biological mutations safely without lab experiments.
- •For biotech safety officers: Evaluate AI tools augmented with biological design software to identify and mitigate potentially dangerous pathogen engineering.
Authors
Yuzhen Wang, Chunxiao Li, Baichen Le, Lewen Yan, Jie Zhang, Yucheng T. Yang, Xing-Ming Zhao, Lijun Li, Jing Shao
Abstract
As large language models (LLMs) acquire advanced biological design capabilities, they offer substantial research benefits but simultaneously introduce severe dual-use risks, such as engineering enhanced pathogens or drug-resistant bacteria. While biosecurity frameworks recognize these threats, the actual risk of LLMs executing dangerous biological modifications remains unquantified, particularly when augmented with specialized biological design tools (BDTs). Existing evaluations primarily assess hazardous knowledge possession rather than practical application, overlook risk amplification from BDTs, and rely on unreliable, expensive, or hazardous validation methods. To address these gaps, we introduce DRYLAB-Bench (Design Risk Yielded by LLM Agent in Biodesign), the first comprehensive benchmark evaluating the high-risk biological mutation design capabilities of LLM agents. Moving beyond basic question-answering, DRYLAB-Bench tasks agents with 30 genuinely hazardous mutation-design scenarios distilled from biosecurity guidelines. It evaluates models in a multi-round, tool-augmented environment where LLMs orchestrate BDTs to iteratively refine candidates. Crucially, it scores designs directly against 187,695 empirical wet-lab measurements from ProteinGym and ViroGym using our proposed Biosecurity Risk Score (BRS), quantifying risk-associated effects while enabling scalable and safe ground-truth verification. Evaluations of 15 frontier models reveal substantial baseline threats: without tools, 67.6% of one-shot designs exceed the average measured variant, and 21.4% reach the highest-risk decile. Equipping models with BDTs significantly amplifies this risk, raising the pooled risk by \(ΔBRS=+0.116\) and the top-decile share to 23.5%. Code: https://github.com/drylab-bio/DRYLAB-Bench