Minkowski positional encoding improves spatiotemporal perception in models
MinkowskiPE: Minkowski Positional Encoding for Spatiotemporal Perception
Machine LearningArtificial IntelligenceComputer Vision and Pattern Recognition
Summary
Understanding how things change over time and space at the same time is important but complicated. The authors created a new way to help computer models pay attention to both space and time together using a special math idea called Lorentz transformations. This new method makes the models better at predicting molecular movements and video frames with fewer parameters. It also keeps the model easy to use with existing technology.
What this means in practice
- •For computational chemists: Improve predictions of molecular movement in simulations using explicit spatiotemporal encoding for more accurate trajectory modeling.
- •For video technology teams: Reduce error in video frame prediction with fewer model parameters by incorporating spacetime-aware positional encoding in neural networks.
Authors
Yuhao Li, Louie Hong Yao, Tianyi Shi, Hanqun Cao, Hongxia Hao, Zhen Zhao, Shengchao Liu
Abstract
Modeling spatiotemporal coupling is a key challenge in building physical intelligence across scales, from microscopic to macroscopic. Existing models capture such structure broadly through physics-motivated dynamical formulations or learning-motivated architectures. The former provide stronger priors but may constrain flexibility, whereas the latter are more flexible but leave the spatiotemporal coupling largely implicit. We therefore seek an approach that combines flexible learning with an explicit geometric bias for jointly modeling time and space. To this end, we propose Minkowski Positional Encoding (MinkowskiPE), which uses joint temporal and spatial coordinates to parameterize Lorentz transformations applied to query and key features. With MinkowskiPE, the query-key attention score depends on position only through the relative spacetime displacement between the two tokens and is therefore invariant to global translation of the coordinates. This paradigm retains the standard dot-product attention interface and remains compatible with efficient attention implementations. We evaluate MinkowskiPE on microscopic molecular dynamics and macroscopic video prediction tasks, achieving the best results on all nine multi-trajectory molecular evaluations and reducing KTH video-prediction MSE by 9.9% relative to the best baseline while using roughly one-tenth as many parameters.