Structured retrieval improves quantitative questions in perovskite research
Benchmark for Structured Retrieval-Augmented Generation in Perovskite Photovoltaics
Machine LearningArtificial IntelligenceInformation Retrieval
Summary
Finding precise answers from scientific papers about solar cell materials is hard because it involves numbers from many studies. The authors show that simply searching text passages gives low accuracy in answering quantitative questions. Instead, organizing the data into tables and letting the AI run database queries on them greatly improves accuracy. This method makes it possible to handle complex questions about measurements from thousands of experiments, something pure text search struggles with.
What this means in practice
- •For materials science engineers: Access aggregated experimental data about perovskite solar cells with higher accuracy by querying structured device information instead of plain text.
- •For database developers: Build enhanced literature search tools that use structured tables and query execution to answer complex quantitative scientific questions.
Authors
Qiwei Han, Chi Zhou, Gaoxiang Yuan
Abstract
While retrieval-augmented language models increasingly serve as literature assistants in the physical sciences, existing benchmarks evaluate them primarily on qualitative text overlap or through model-based scoring rather than against verified experimental ground truth. In experimental materials disciplines such as perovskite photovoltaics, however, the most critical inquiries are inherently quantitative and aggregate measurements across dozens or hundreds of papers. Here, we present a quantitative question-answering benchmark where all 755 ground-truth answers are computed directly from 45,441 measured device records (40,889 human-curated records through 2021 and 4,552 records extracted from 2022-2026 literature). Benchmarking six retrieval architectures over 158,274 passages from 25,662 papers using an open 14B model on a single 24 GB GPU, standard passage retrieval tops out at 16-23% accuracy (compared to 1.6-6.5% zero-shot). In contrast, equipping the language model with read-only SQL execution over the structured device table elevates accuracy to 51.5% on historical and 50.8% on prospective literature -- a 33.01.8 percentage-point advantage over hybrid text retrieval. Aggregate queries spanning extrema, distributions, and filtered record counts prove virtually unsolvable via text retrieval alone, demonstrating that systematic table extraction is an operational prerequisite for literature-scale quantitative inquiry.