Expressivity of Shallow Neural Networks Over Finite Fields
Neural and Evolutionary Computing
Summary
The authors explore how well shallow polynomial neural networks (PNNs) with simple monomial activations can express functions when working over finite fields (algebraic systems with a finite number of elements). They define a concept called a neuromanifold that captures all outputs a network can produce from its weights, and measure expressivity by counting how many distinct outputs exist. By relating this counting problem to deep mathematical results like the Weil conjectures, they establish upper and lower bounds on expressivity. They also show that the field's characteristic (a property of the number system) significantly impacts the network’s expressivity.
Authors
Maksym Zubkov, Carol Wu, Shiwei Yang, Param Mody, Yifei Chen
Abstract
We study the expressivity of shallow polynomial neural networks (PNNs) with monomial activation functions over finite fields. For a given architecture, we define a neuromanifold as the image of the map from all possible network weights into the product of polynomial rings. We quantify the expressivity by the cardinality of the neuromanifold, and derive a natural lower and upper bound. This leads to counting rational points over finite fields, a problem closely linked to the Weil conjectures. Finally, we present an architecture that exhibits a striking difference in the neuromanifolds when considered over a characteristic zero versus a finite-characteristic field, illustrating the critical role of field characteristic in the notion of expressivity.