Rational approximations speed up multilayer perceptron inference with good accuracy
Rationally Linearized MLPs
Machine Learning
Summary
Multilayer perceptrons (MLPs) use complex math functions called activations, which can make them slow to use. The authors found a way to approximate these functions with simpler rational functions that need less math but still work well. This lets computers run MLPs faster while keeping accuracy high. They showed how this method helps in areas like 3D modeling and language understanding.
What this means in practice
- •For 3d graphics developers: Reduce computational cost when running implicit 3D shape models by using linearized MLPs based on rational approximations.
- •For large language model engineers: Lower the inference cost of large language models by integrating rationally linearized MLPs without losing prediction quality.
Authors
Arijit Sehanobish, Byeongchan Kim, Avinava Dubey, Min-hwan Oh, Krzysztof Choromanski
Abstract
We present Rationally Linearized Multilayer Perceptrons(RaLiMPs), a new approach to efficiently linearizing MLPs using low-degree rational approximations of standard activation functions. Inspired by polynomial-based MLP linearization, we show that rational approximations can provide accurate representations at substantially lower degree. RaLiMPs exploit these approximations to construct kernel representations with narrow bottlenecks, enabling efficient composition across MLP layers. We provide theoretical guarantees for the resulting approximation and evaluate RaLiMPs across applications ranging from implicit 3D models to large language models. Our experiments demonstrate favorable accuracy--efficiency tradeoffs, with RaLiMPs achieving competitive or improved accuracy while substantially reducing inference cost.