Tensor decoupling made more reliable with less manual tuning
Making Tensor Decoupling More Robust and Less Parametric
Machine Learning
Summary
Sometimes, computers try to simplify complex multivariable functions by breaking them down into simpler parts, but current methods can get confused if some outputs are much bigger than others or if settings need careful adjustment. The authors found ways to fix these issues by first adjusting the inputs to avoid bias toward larger numbers and then by using a smarter way to control curve fitting that doesn’t need much manual tuning. They also shared their improved method in an easy-to-use software package for other developers.
What this means in practice
- •For machine learning engineers: Improve model components that represent complex functions with less manual tuning and more robust performance across varying data scales.
- •For scientific computing teams: Use the method to extract simpler functional representations from simulation data without manual spline parameter selection.
Authors
Maxime Rochkoulets, Joppe De Jonghe, Mariya Ishteva
Abstract
Tensor decoupling algorithms recover compact representations of multivariate functions by combining linear factor matrices with univariate internal nonlinearities. We identify two limitations of recent methods. First, their performance is not invariant to mismatched output or gradient magnitudes, and second, their spline internals require manual tuning of their degrees of freedom. We propose two lightweight fixes. The first is a preprocessing step that rescales the input Jacobian tensor, removing both the bias towards larger magnitude outputs and the ill-conditioning of certain factor updates caused by such a mismatch. The second replaces B-spline internals by penalized splines coupled with generalized cross-validation, making the method less dependent on the user selecting the right number of control points. Finally, we release an efficient open-source JAX implementation of our algorithm.