Hardness of learning functions linked to noise masking with channels
Universal Hardness From Masking Channels
Machine LearningComputational Complexity
Summary
The paper studies how hard it is to learn or predict the outputs of complex functions by adding noise that hides their patterns. The authors connect a classic idea about hard-to-compute functions with a way of injecting randomness to make outputs appear random. They develop a general method that works for many types of functions and learning goals, showing that the difficulty of learning matches how much noise is needed to conceal the function’s structure. This approach also ties to ideas in machine learning and information theory, giving a broad framework for understanding when and why certain learning problems are hard.
What this means in practice
- •For machine learning engineers: Design better algorithms by understanding limits of learning due to intrinsic function hardness masked by noise.
- •For cryptography developers: Build stronger cryptographic primitives by applying universal hardness guarantees from noise-based function masking.
A theory result. No direct application yet.
Authors
Parikshit Gopalan, Manoj Prabhakaran, Pranay Tankala
Abstract
We prove a broad learning-theoretic generalization of the celebrated hardcore lemma concerning the average-case complexity of computing a Boolean function (Impagliazzo, 1995). Our starting point is the observation that the classic notion of a hardcore measure, which reweights a function's input distribution, is equivalent to an input-dependent masking channel that injects randomness into the function's output distribution until it appears fully pseudorandom. The analogue of density of the measure is the fidelity of the channel. From this viewpoint, we develop a general framework for certifying hardness that allows for non-Boolean functions $f$ and a wide range of notions of hardness besides the classic 0-1 loss, by choosing the right notion of fidelity via convex duality. We show the difficulty of learning $f$ using a class $\mathcal{C}$ to minimize a convex loss is tightly characterized by the maximum fidelity possible for a masking channel, which measures the least amount of noise the channel must add to hide the structure of $f$ and make it $\mathcal{C}$-indistinguishable from random noise. Using our framework, we derive a number of multiclass generalizations of the hardcore lemma, and reveal some unexpected subtleties in going to larger alphabets. We construct a universal masking channel that applies to many natural notions of hardness simultaneously. Such channels are dual to omnipredictors, which optimize multiple losses at once (Gopalan, Kalai, Reingold, Sharan, and Wieder, 2022). We observe a symmetry between masking channels and generalizations of the hardcore lemma using pseudoentropy. We present a tight characterization of pseudoentropy using the MaxEnt algorithm from machine learning. A long line of research has shown that learning and hardness are two sides of the same coin; our framework gives a crisp and general formulation of this principle via convex duality.