Deep representation learning reveals cost of feature selection

Brownian Heads for Deep ReLU Representations: Activation Mass and the Cost of Same-Sample Selection

Machine Learning

Summary

Selecting and using the same data to pick hidden features and make predictions can hide some hidden costs in deep neural networks. The authors study a complex measure called Rademacher complexity to understand these costs for networks using a common ReLU function. They connect this complexity to how active or 'used' the network features are, calling this activation mass. Their mathematical results help separate the difficulty of choosing features from how big the chosen features are, clarifying how selection impacts learning. Experiments show their bounds are accurate and reveal gaps in understanding selection costs.

What this means in practice

  • For machine learning engineers: Estimate the hidden cost of selecting neural network features and improve model risk assessment.
  • For data scientists: Better understand the trade-offs when using the same data for feature selection and model training to enhance prediction reliability.

A theory result. No direct application yet.

Authors

Mahdi Mohammadigohari, Nicole Mücke

Abstract

Deep representation learning often selects hidden features and fits the final predictor on the same sample, so fixed-feature analysis performed after selection can omit selection cost. We study the conditional empirical Rademacher complexity of deep ReLU representations followed by bounded-norm predictors in additive or Lévy-Brownian RKHSs, termed Brownian heads. For a fixed representation, we derive an exact dual identity and sharp bounds in terms of activation mass, the average norm of the observed hidden vectors. Under same-sample selection, the representation supremum induces a quadratic Rademacher process. Brownian layer-cake and Gaussian-projection identities reduce it to coordinatewise or signed projected threshold traces, separating realized scale from selection complexity. For samples with pairwise-distinct inputs, explicit scalar ReLU families match the finite-trace and VC rates up to universal constants at the realized trace-and-envelope level. Induced-norm contraction also yields architecture-level bounds for rectangular, rank-deficient ReLU networks. Experiments verify the sharp bounds and rates, exhibit a selection gap at fixed activation mass, and assess the predictive feasibility of Brownian heads.
View PDFOpen arXiv