Ai agents struggle to exactly reconstruct sequencing workflows and molecular structures

Evaluating and improving AI agents for sequencing protocol understanding

Artificial Intelligence

Summary

Sequencing protocols describe how to prepare molecules for genetic analysis, and it's important to understand exactly how molecules are arranged and processed. The authors created a new test called LibStructBench to see if AI agents can accurately rebuild these molecular structures and workflows from documents. They found that while current AI can roughly guess the setups, they often make mistakes in important details like molecular states and connections. The study shows that just changing the underlying AI language model affects performance more than changing the way the AI is organized. Adding extra knowledge helped in some small ways but didn’t improve overall workflow understanding.

What this means in practice

Authors

Chi-Lam Poon, Michael Goneos, Zezheng Song, Xi Chen

Abstract

Existing scientific-agent benchmarks rarely evaluate whether agents can reconstruct the exact molecular structures and generation workflows specified by sequencing protocols, a capability critical for analyzing customized sequencing assays. We introduce LibStructBench, a document-grounded benchmark for reconstructing sequencing-library structures and molecular workflows. Our experiments show that frontier agents often produce approximately correct reconstructions but remain unreliable at exact reconstruction, particularly in molecular-state representation and workflow connectivity. Switching the underlying language model produced larger average performance differences than switching agent harnesses, with a statistically significant difference only for molecular-state reconstruction. Additional knowledge improved oligo-family recovery in a small pilot without improving mean molecular-workflow scores. Autonomous accumulation of procedural rules and cross-protocol examples showed non-monotonic validation performance and no mean held-out molecular-workflow improvement in the tested trajectory. These findings highlight the need for more controlled and verifiable approaches to scientific reasoning about molecular processes in complex experimental protocols.