How neural network depth affects precision limits and accuracy loss

Depth Laws for the Precision Floor of Trained Neural Networks: Amplification, Residual Scaling, and a Quantization-Aware Training Paradox

Machine Learning

Summary

This paper studies how the number of bits used to represent neural network computations affects their accuracy, especially as networks get deeper. The authors find that deeper networks require higher precision because small errors get amplified. They also discover that certain network designs and training methods influence how much precision is needed. Surprisingly, a method called noise-aware training helps shallow networks tolerate errors better but becomes less effective as networks grow deeper. These results help explain the limits of reducing precision in various types of neural networks without losing too much accuracy.

What this means in practice

  • For machine learning engineers: Optimize neural network bit-width settings to balance accuracy and resource use when deploying models of different depths in production.
  • For hardware design teams: Design quantization strategies that consider network depth and residual scaling to improve efficiency of AI chips without large accuracy loss.

Authors

Ahmad S. Tarawneh

Abstract

How many bits does a network need before its accuracy collapses, and how does this grow with depth? We study the precision floor, the perturbation level or bit-width at which accuracy falls halfway to chance, in MLPs, CNNs, Vision Transformers and nine pretrained language models, under post-training quantization (PTQ) and quantization- or noise-aware training (QAT). (i) A first-order theory sets the floor through one full-precision quantity, the predictive amplification $G$: $η_c=Λ/G$, and $G^2$ grows linearly in depth at a rate proportional to the squared residual branch scale. (ii) The predicted equality $α_{PTQ}=ρ$ of depth exponents holds within 95% intervals in twelve of thirteen trained architectures and in GPT-2 from 12 to 48 layers, with $Λ=1.45\pm14\%$ across trained architectures. (iii) Residual branches scaled by $1/\sqrt{D}$ and pre-normalisation remove the depth penalty, and each quantizer turns noise into bits at a rate fixed by its step rule, giving $b_c=(α/γ)\log_2 D+C$. (iv) A QAT paradox: noise-aware training roughly doubles the tolerable noise of shallow networks, but the gain decays with depth, so the depth law steepens ($α_{QAT}/α_{PTQ}=1.45$-$1.47$ on two datasets, ten seeds each). Decision margins, cross-layer error cancellation and heavy tails do not set the floor.
View PDFOpen arXiv