Rank and fast computation of relu network pathlifting jacobian
Rank and computation of the pathlifting Jacobian of a DAG ReLU network
Machine Learning
Summary
Deep neural networks with ReLU elements use complex math to learn from data, but calculating certain derivatives is costly. The paper explains a simpler way to find the rank and compute a key matrix called the pathlifting Jacobian by breaking down the network step-by-step. The authors connect the network’s structure to a special sparse matrix and show how to efficiently compute this matrix without the usual slow methods. They also provide code that tests how their approach speeds up computations in practice.
What this means in practice
- •For machine learning engineers: Compute network derivatives faster without backpropagation for training and sensitivity analysis.
- •For signal processing developers: Use efficient Jacobian computations in network-based signal models to improve real-time processing speed.
Authors
Manon Verbockhaven
Abstract
This paper provides a self-contained proof of the rank of the pathlifting Jacobian of a DAG ReLU network by performing an induction on the network's number of hidden nodes. In fact, the induction is elementary, and the key recipe is to consider the skeleton matrix of the network, a sparse matrix encoding the network paths, and transform the representation of one of its hidden neurons into an output node. The proof relies on intermediate propositions which link the pathlifting, its Jacobian, the network parameters, and its skeleton matrix, which, on top of permitting to conclude on the rank of the pathlifting Jacobian, also provide a way to compute it without backpropagation and whose computation cost is super efficient in practice compare to usual backpropagation. The paper is provided with a Python module that implements the different propositions of the paper for feed forward networks and is used to experimentally quantifies the computational gain of computing the pathlifting Jacobian with the proposed theory.