Human movement prediction improves by learning forward and backward representations

BAL: Bidirectional Autoregression in Latent Space for Learning Human Movement Representations

Computer Vision and Pattern RecognitionMachine Learning

Summary

Understanding and modeling human movement is important for many applications, but existing methods often rely on tricky assumptions that can make learning inconsistent. The authors propose BAL, a new approach that learns human motion by predicting what happens next and what happened before in an abstract form instead of raw joint positions. This method uses the natural order of movements and prevents the model from collapsing into meaningless patterns by splitting predictions into overall motion and small local changes. Their experiments show that this approach produces better representations for tasks like describing movements, predicting future poses, and filling in missing motions.

What this means in practice

Authors

Genki Kinoshita, Ko Nishino

Abstract

Learning semantically meaningful human movement representations from 3D pose sequences is essential for human behavior modeling. Existing self-supervised methods derive supervision from artificial assumptions, e.g., invariance to augmentation within a certain range or recoverability of masked regions at a certain temporal scale, introducing heuristic hyperparameters that make the learned semantics sensitive to their settings. We propose BAL, a self-supervised framework that instead exploits supervision intrinsic to human movement: the natural ordering structure of everyday behavior. BAL autoregressively predicts in a semantically abstract latent space, analogous to how humans anticipate others' actions rather than their precise joint configurations. Bidirectional autoregressive prediction, both forward and backward in time, fully leverages these ordering constraints while enabling representation extraction using both past and future context at inference. To suppress representation collapse, BAL decomposes each prediction target into a global component capturing sequence-level semantics and a local component capturing context-independent movement. We prove that the local targets retain nontrivial angular diversity unless the representations are completely collapsed, preserving a meaningful training signal. Experiments on large-scale datasets demonstrate that BAL's representations are highly effective for motion captioning, forecasting, and interpolation, outperforming baselines on most evaluation metrics.