Using natural language to improve predictions from limited sensor data
Generative Priors Conditioned on Natural Language for Bayesian Inversion in PDEs
Machine Learning
Summary
Predicting important physical quantities from noisy measurements is hard when data is limited. This paper shows how combining short text descriptions about the system with a learned model can make these predictions better and provide trustworthy uncertainty estimates. The authors develop two methods that learn from paired text and numerical data to help infer outcomes for new systems using few sensor readings plus descriptions. They test these methods on heat flow, wave behavior, and weather data, finding trade-offs between speed, accuracy, and uncertainty quality.
What this means in practice
- •For engineering simulation teams: Infer system properties more accurately using a few sensors combined with textual experiment notes to speed up modeling in new setups.
- •For weather forecasters: Improve uncertainty estimates for weather models by combining reanalysis data with human observations described in language.
Authors
Pengyu Zhang, Mark Girolami, Arnaud Vadeboncoeur
Abstract
Inferring quantities of interest (QoI) from data is a central task in Science and Engineering. In such contexts, we often have access to both quantitative data and qualitative data. Quantitative data may be represented by noisy sensor measurements, simulation data, re-analysis data; qualitative data may be in the form of text descriptions of experimental setups, expected experiment outcomes, and human-perceived system behaviours. The task we address in this paper is the following. Given a training set of paired qualitative text and quantitative QoI data, we learn to exploit the inherent correlation between the two modalities to learn a highly informative data-driven natural-language-conditional Bayesian prior, such that when presented with a new physical system, we can coherently combine (i) the training dataset, (ii) qualitative text describing the new system, and (iii) a small number of noisy sensor readings from that new system, to perform inference and uncertainty quantification (UQ) over the QoI. To achieve this task, we develop two parallel approaches, one uses conditional diffusion and the other conditional autoencoders, and compare both against classical Bayesian methodology, unconditional generative models and deterministic supervised methods. Each approach has specific strengths and tradeoffs; conditional autoencoder offers theoretical tractability, allows for fast posterior sampling, and provides better-calibrated UQ, whereas conditional diffusion is explored for greater expressiveness and capturing complex posteriors with irregular QoI fields. The approach is tested on the steady-state heat equation, damped Helmholtz equation, and UK weather reanalysis data.