Foundation models enable faster statistical inference with less computing
HIFI: A Highly Efficient Framework for Statistical Inference with Foundation Models
Machine Learning
Summary
Making predictions and understanding data using big AI models can take a lot of computer power, especially when checking how certain those predictions are. The authors created a new method called HIFI that trains a big AI model once on a large dataset, then quickly checks the reliability of results on a smaller set without retraining the entire model. This approach works well with different types of AI models and reduces computing time by up to 92%. It also helps scientists get better confidence when analyzing complex data like gene expression.
What this means in practice
- •For machine learning engineers: Train large AI models once and perform multiple uncertainty checks efficiently without retraining the whole model each time.
- •For biomedical data teams: Get reliable confidence estimates when using AI models to analyze gene-expression data with much less computational cost.
Authors
Po-Wei Chen, Shan Yu, Chunlin Li
Abstract
Foundation models provide powerful representations for unstructured data, such as text and images, and parameter-efficient fine-tuning (PEFT) makes their adaptation to downstream tasks increasingly practical. For neural estimators that lack a tractable sampling distribution, resampling-based procedures offer a flexible route to statistical inference, but they often require hundreds of model fits, making them computationally prohibitive at foundation-model scale. We introduce HIFI, a two-stage fine-tuning framework for statistical inference on parameters such as conditional means. Stage 1 adapts a foundation model once on a larger sample, capturing the complex structure of the data and controlling estimation bias through a single fit. Stage 2 applies a resampling inference procedure to a smaller independent sample, concurrently fitting multiple lightweight PEFT estimators to the residual to quantify estimation variability without relearning the full task. HIFI accommodates a broad class of resampling procedures, including ensemble subsampling and the nonparametric bootstrap. Across vision and text experiments, HIFI scales to pretrained models ranging from tens of millions to more than ten billion parameters while providing accurate point estimates and well-calibrated confidence intervals across diverse architectures, and reduces training compute by up to 91.9% relative to direct resampling. An application to gene-expression inference illustrates the practical value of the framework for uncertainty-aware scientific inference.