Understanding Deep Learning via Entropy Space Theory
Artificial Intelligence
Summary
The authors introduce a new way to understand deep learning called entropy space theory, which uses a mathematical framework to describe all possible models without relying on specific parameters. They show that this space follows formal rules like those in normed spaces, which help measure and compare models. Using this theory, they propose a system to label and rank different model states based on information entropy. This provides a foundational approach to study deep learning mathematically.
deep learningentropynormed spacetopological structureinformation entropycoordinate systemmathematical frameworkmodel stateaxiomatic framework
Authors
Li Li, Tong Zhang, Wentao Yu, Zuobin Wang
Abstract
Deep learning is often criticized for its theoretical research lagging behind practice. To make deep learning easier to understand, the entropy space theory is first introduced here. The entropy space can cover all the possibilities of any deep learning model by topological structure. It is independent of network parameters. Through the designed fundamental operations and norm, entropy space is proven to be a normed space within the formal axiomatic framework. Based on the theory, a unified coordinate system is proposed. It can coordinatize every state of a model and rank them by compression of the maximal value of information entropy. The theory offers a novel priori framework for mathematical fundamentals of deep learning.