Continuous optimization enables machine learning with p-adic numbers

Continuous Optimization for p-adic Models

Machine Learning

Summary

Machine learning models usually use regular numbers, but some tasks need special number systems called p-adic numbers, which are tricky because they don’t behave like normal ones. The authors found a way to do smooth optimization with p-adic numbers by using a special mathematical space that makes them behave more like normal numbers. This method helps train models that can perform tasks, like modular arithmetic, which regular models cannot do easily. Their approach also works with common training techniques like momentum and Adam.

What this means in practice

Authors

Julian Salazar, Dimitri Kanevsky, Matt Harvey, Pascal Getreuer, Lucas Dixon

Abstract

We present the first method for native, continuous gradient descent for machine learning models with $p$-adic parameters. Existing native optimizers are discrete, mostly combinatorial searches, as the $p$-adic numbers $\mathbb{Q}_p$ are totally disconnected, with standard losses that are flat away from their minima. To enable continuous optimization, we propose working with $\mathbb{Q}_p$ via its Berkovich affine line: a canonical, path-connected expansion of $\mathbb{Q}_p$ that preserves its isometries and uniquely extends its analytic maps. This hull is a metric tree with interpretable points and local derivatives, which we show enables effective optimizers and backpropagation. We formulate gradient descent and show that its approximations efficiently learn linear models with coefficients in $\mathbb{Q}_p$ to do modular arithmetic, an XOR-like task not expressible by linear models in $\mathbb{R}$. We also demonstrate momentum and Adam variants, linear regression, and classification on binary-encoded hierarchies (Quillian semantic networks), addressing open problems posed by Martins (2025). Library at https://github.com/google-deepmind/padic-ml
View PDFOpen arXiv