Functional Attention: From Pairwise Affinities to Functional Correspondences
2026-05-29 • Machine Learning
Machine Learning
AI summaryⓘ
The authors present a new way to improve how machines learn relationships between complex functions, which is important in many scientific and engineering problems. Instead of treating continuous data as a bunch of separate pieces, their method called Functional Attention looks at the whole structure at once using math ideas from geometry. This helps make their models work well even when the data is sampled differently or at different resolutions. They tested it on tasks like solving equations and 3D object recognition and found it performs very well.
operator learningfunctional attentiontransformertoken-wise attentionfunctional mapslinear operatorsresolution invariancepartial differential equations (PDEs)3D segmentationmachine learning
Authors
Jiefang Xiao, Maolin Gao, Simon Weber, Guandao Yang, Daniel Cremers
Abstract
Learning mappings between infinite-dimensional function spaces, or operator learning, is essential for many machine learning applications. Although transformer-based operators are popular, they often rely on token-wise attention. These methods treat continuous fields as discrete tokens and usually ignore the global functional structure. We introduce \emph{Functional Attention}, which reinterprets attention as a functional correspondence between adaptive bases. Inspired by geometric functional maps, our method replaces softmax affinities with structured linear operators. This yields a compact, generalizable, resolution-invariant representation that explicitly captures global dependencies. Experiments demonstrate that \emph{Functional Attention} can match state-of-the-art performance in many operator learning tasks, including solving PDEs, 3D segmentation, and regression, while remaining robust to varying discretizations. Project page is available at https://github.com/xjffff/FUNCATTN.