Scientific agents need memory that knows when facts matter most
The Right Fact, the Wrong Decision: SciValidityBench for Scientific Agent Memory
Artificial Intelligence
Summary
Most systems that remember past events focus on whether they can recall the right information. But scientific decision-making is trickier because even a true fact might not be useful depending on the circumstances. The authors created SciValidityBench, a testing setup that checks if agents not only remember facts but also understand when those facts should influence decisions based on scientific rules. Their tests across multiple AI models showed that encoding the scientific context improves decision-making better than just remembering facts alone.
What this means in practice
- •For ai system developers: Create scientific AI agents that decide based on memory plus context about when facts are valid to use during experiments.
- •For quality assurance teams: Design tests for AI models ensuring they understand validity conditions, not just recall experimental data.
Authors
Yuexi Pan, Yongchao Chen
Abstract
Most agent-memory benchmarks frame long-term memory as retrieval: can an agent recover the relevant record from a long and evolving history? Scientific agents face a stricter problem: a correctly remembered fact may still be invalid to use. The same reported test result can support confirmation when the outcome and stopping rule were fixed in advance, yet only exploration when the outcome was selected from twenty inspected results. We call this scientific use validity: whether remembered evidence is eligible to influence a decision under the state governing its interpretation. We introduce SciValidityBench, an executable benchmark factory with 64 TaskSpecs spanning five validity modules and six scientific domains. TaskSpecs couple generated experimental histories with scientific-state definitions, programmatic decision oracles, controlled interventions, and automatic scoring. The SciStateForge compiler-reader framework turns distributed histories into explicit decision-governing state. Counterfactual pairs and targeted corruption-repair controls then test whether models use those dependencies and localize failures. Across four model families, compiled scientific state improves valid decision-making by 9.2-18.8 percentage points over a generic Event Graph. Retrieval alone does not reproduce this cross-family gain, while content-identical flat and typed views show that the improvement comes from exposing governing state rather than a particular graph serialization. SciValidityBench thus shifts memory evaluation from finding the right fact to deciding whether that fact should control the decision.