Active feature acquisition improves with synergy in some low budget cases
When Does Synergy Help Active Feature Acquisition? A PID-Based Study
Machine Learning
Summary
Selecting which features to use in data analysis can be costly, so systems try to pick the best ones without wasting resources. The authors study when combining information from pairs of features helps make better selections. They find that synergy between features boosts performance mostly when budgets for collecting features are small, but this advantage fades as budgets grow. Their work also shows challenges in turning this synergy insight into consistent practical gains.
What this means in practice
- •For medical data analysts: Improve feature selection for patient monitoring systems when only few measurements can be taken, using synergy to prioritize informative data.
- •For machine learning engineers: Design feature acquisition processes that consider pairwise synergy to enhance model predictions under strict feature budgets in tabular data contexts.
Authors
Jie Li, Maruf A. Dhali, Hjalmar R. Bouma
Abstract
Active feature acquisition (AFA) sequentially selects informative features under budget constraints. However, existing policies rarely distinguish whether information contributed by interacting features is redundant, unique, or synergistic. We introduce SynAFA, a state-dependent AFA policy that combines pairwise joint information and conditional information, with Partial Information Decomposition (PID) characterizing its information structure. Across five tabular datasets and MNIST-loop, performance is heterogeneous. SynAFA shows its strongest gains at low budgets on PhysioNet, where synergistic and redundant feature pairs are supported by permutation tests, but its advantage diminishes as budgets increase and does not depend on pair proposals. SynAFA performs significantly worse than nearly all baselines on MiniBooNE, and than CAE on MNIST-loop. Controlled synthetic experiments further show that, within budget, SynAFA's advantage rises as joint information becomes more synergy-dominated, including when total joint information is held approximately constant. Further analyses show that the budget-dependent erosion is not resolved by non-greedy local search, which improves a diagnostic set-level objective but leaves predictive performance no better, often significantly worse, and that improvements in this objective are weakly aligned with the fixed classifier's predictive utility. These findings characterize when pairwise synergy can benefit AFA while exposing a persistent challenge in translating local information into effective acquisition objectives.