Distributed feature extraction complements side information to improve estimation accuracy
Universal Distributed Feature Extraction Complementary to Side Information via H-Score
Information TheoryMachine Learning
Summary
This paper looks at how devices on the edge of a network can send useful summarized data to a central place that already has some extra related information. The challenge is to design local data summaries that work well for any future purpose while not repeating what the center already knows. The authors show how to mathematically break down the joint information and learn the best summaries that avoid redundancy and do well broadly. Their method performs nearly as well as if the summaries were specially designed for each task, as long as enough data features are shared.
What this means in practice
- •For edge computing engineers: Design feature extraction algorithms sending small summaries that efficiently complement existing data at a central server for diverse estimation tasks.
- •For sensor network operators: Implement data compression methods that avoid redundant information transmission when central units have related auxiliary data for better remote sensing accuracy.
Authors
Yuan Xinjie, Dier Tang, Shaolun-Huang
Abstract
This paper considers a distributed estimation system where edge nodes transmit averaged features to a center node for parameter estimation. A key challenge arises when the center node possesses auxiliary side information: how to design local extractors that are universally optimal for any downstream task, while avoiding redundancy with the side information. To tackle this problem, we orthogonally decompose the joint function space and demonstrate in the weak-dependence regime that the modal decomposition of the canonical dependence kernel's conditional-additive component yields optimal, task-independent features. We construct a distributed nested H-score to learn this decomposition and show that, under a limited feature budget, the mean-squared error (MSE) gap to the best task-specific features is determined solely by the task's alignment with the discarded, least informative modes, and vanishes to leading order for every square-integrable task once the budget is sufficient.