Feature transformation learning improves tabular data predictions
Hierarchical and Permutation-Invariant Feature Transformation Learning via Policy-Guided Embedding Search
Machine LearningArtificial Intelligence
Summary
Improving predictions from spreadsheet-like data often means turning messy raw info into clearer forms. The paper introduces a way to better learn how these transformations work by respecting the natural hierarchy in data features and ignoring irrelevant order in operations. It also uses a smart trial-and-error search method that balances how accurate and efficient the transformations are. Their approach showed stronger and more reliable results compared to other methods on several data sets.
What this means in practice
- •For data science teams: Create better feature transformations that boost predictive models for tabular data by using hierarchy-aware and permutation-invariant learning.
- •For business analysts: Improve decision-making tools by efficiently generating accurate data features without manual trial-and-error.
Authors
Rui Liu, Tao Zhe, Yanyong Huang, Sankha Narayan Guria, Xiao Luo, Wei Fan, Yanjie Fu, Dongjie Wang
Abstract
Feature transformation improves predictive performance on tabular data by constructing informative abstractions from raw features. Recent generative approaches encode transformation knowledge into continuous embedding spaces for efficient exploration of candidate strategies, but face three key limitations: (1) overlooking hierarchical relationships between low-level features, operations, and high-level abstractions; (2) enforcing order-sensitive embeddings on inherently permutation-invariant transformation sequences, thereby introducing systematic bias; and (3) relying on gradient-based search, which is ill-suited to non-convex transformation spaces. We propose a framework with two complementary components. First, a permutation-invariant hierarchical module captures interactions across features, operations, and abstraction levels, with a self-attention pooling mechanism that maps semantically equivalent structures to consistent embeddings aligned with downstream performance. Second, a policy-guided multi-objective reinforcement learning strategy initializes the search from empirically strong seeds and jointly optimizes predictive accuracy and transformation efficiency. Extensive experiments on diverse tabular benchmarks demonstrate the effectiveness and robustness of our framework against strong baselines. Our code and data are publicly available at: https://github.com/RayLiu1103/PHER.