Active feature acquisition policy evaluation improved by separating uncertainty types

Evaluation of Active Feature Acquisition Policies with Tabular Foundation Models

Machine Learning

Summary

Deciding which features of data to gather next can help predict outcomes better but is tricky when relying on limited prior data. The authors show that traditional methods of measuring uncertainty often mix up unknown information caused by little data and randomness in the features themselves. They propose focusing on the uncertainty that reflects real unpredictability in the features, improving how policies for collecting data are evaluated. Their tests show this approach more accurately predicts the value of acquiring features and helps choose better strategies.

What this means in practice

  • For data scientists: Evaluate feature acquisition strategies more accurately when training models from limited offline datasets.
  • For medical data analysts: Improve decisions on which medical tests to order next by better distinguishing uncertainty types in patient data acquisition.

Authors

Yuta Kobayashi, Divyam Madaan, Shalmali Joshi

Abstract

Active feature acquisition learns policies that sequentially acquire features to maximize information about a target variable. We study how to learn and evaluate such policies from finite offline data using prior-data fitted networks (PFNs), which are off-the-shelf models that output posterior predictive distributions without task-specific training. We show that under the imbalanced coverage of offline data, using total predictive entropy as a reward creates an epistemic bias that penalizes acquiring sparsely observed features. Specifically, this reward conflates epistemic uncertainty (arising from lack of offline data) with aleatoric uncertainty (arising from uninformative features). To address this, we target the posterior expected (aleatoric) entropy instead of the total predictive entropy output by a PFN for evaluating feature acquisitions. Empirical evaluations on synthetic and real-world datasets demonstrate that our approach consistently reduces value estimation bias and yields credible intervals with strong empirical coverage, which can translate to improved downstream policy selection.