Integer only method reduces transformer accuracy loss under quantization
TR-PTQ: High-Accuracy Integer-Only Transformer Post Training Quantization via Taylor Region Reformulation
Machine Learning
Summary
Transformers are powerful AI models but hard to make run fast and cheap on small devices because of tricky math steps. The authors show that some parts of the math, like certain layers and functions, cause most of the errors when using simpler number formats. They created a new way to do all the needed calculations using only whole numbers, avoiding complicated decimal math. This keeps the model accurate while making it run efficiently on hardware without special number-handling units.
What this means in practice
- •For mobile app developers: Run transformers efficiently on mobile devices without floating-point hardware by using integer-only computations.
- •For embedded systems engineers: Deploy transformer models on low-power hardware lacking floating-point units with minimal accuracy loss.
Authors
Eliyahu Levy, Adam Teman, Yoni Pugachov
Abstract
Post-training quantization (PTQ) enables efficient deployment, yet transformer architectures remain challenging to quantize due to nonlinear layers. While existing methods attribute accuracy loss to insufficient numerical precision, often necessitating floating-point fallbacks, we demonstrate that degradation is actually driven by specific structural error sources. We find that learned scale parameters in normalization layers and compounded approximations in GELU are the primary error contributors, whereas SoftMax remains inherently robust to aggressive quantization. To address these bottlenecks, we introduce TR-PTQ, a unified integer-only formulation using shared Taylor Region (TR) exponential and logarithm primitives. This approach allows computationally expensive operations, including division and square roots, to be performed entirely in the log-domain via standard integer arithmetic. Combined with a calibration-free, outlier-aware optimization for LayerNorm parameters, our method eliminates the need for floating-point hardware units for nonlinearities, achieving less than 1.5\% absolute accuracy degradation across vision and language benchmarks.