Language models improve domain-specific fusion science reasoning skills
Assessing Language Models for Scientific Applications: Inertial Confinement Fusion as a Use Case
Artificial Intelligence
Summary
Evaluating how well large language models can help with specialized science problems is challenging. The authors tested models on questions about inertial confinement fusion, a complex physics field, using an expert-made quiz covering reasoning and calculations. They found specialized training improves performance but does not guarantee error-free scientific answers. Adding information retrieval helped smaller models but not the most advanced ones. The study highlights that expert checks remain important before trusting AI in scientific roles.
What this means in practice
- •For scientific software developers: Use specialized language models to assist in generating and checking physics equations and reasoning in fusion research software workflows.
- •For technical content creators: Employ domain-adapted language models to improve accuracy in producing educational and technical materials on advanced physics topics.
Authors
P. B. Radha, D. D. Meyerhofer, M. Bhattarai, B. Cheng, L. Diaz, C. L. Rousculp, A. G. Seaton, G. Gupta, P. Gurumurthy, V. Gutta, A. Jhunjhunwala, A. Vem, P. Cook, G. Rowe
Abstract
Advancing language models for specialized scientific applications requires evaluations of scientific fluency, physical reasoning, and quantitative analyses. Using inertial confinement fusion (ICF) as a test case, we have developed an expert-curated benchmark of 250 questions spanning qualitative reasoning, numerical calculation, symbolic derivation, and equation formulation. Responses have been evaluated using deterministic numerical and symbolic checks together with language-model judges. We compared a variety of out-of-the-box closed-source and open-source large language models with Prospero, a model specialized through combined domain-adaptive pretraining and supervised fine-tuning for ICF. We also conducted initial retrieval-assisted evaluations. Prospero improved its aggregate score by 10.7 percentage points over its base model, with gains across all question types. Frontier models achieved the highest aggregate scores, but critical physics errors persisted, particularly in numerical calculations. We note that domain-specific performance can improve through specialization, but benchmark gains alone do not establish scientific reliability. Retrieval improved performance for a smaller model but yielded no statistically significant improvement for a frontier model. Two scientific stress tests further illustrate useful capabilities and the need for expert assessment of physical assumptions and model validity. Based on these findings, we recommend evaluating existing domain capabilities and retrieval benefits before committing to substantial additional specialization.