Learning Probabilistic Embeddings for Unsupervised Action Segmentation
2026-07-06 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors focus on breaking down long videos into meaningful action parts without using labeled data. Earlier methods used fixed (deterministic) ways to represent each video frame, which can get stuck and not improve much over time. Instead, the authors propose representing each frame with a range of possible values (probabilistic embeddings) using Gaussian distributions, which helps avoid getting stuck and improves learning. Their method performs better or equally well compared to previous ones on multiple datasets, with noticeable improvements in key accuracy metrics.
unsupervised learningtemporal action segmentationoptimal transportpseudo labelsprobabilistic embeddingGaussian distributionframe representationgradient descentMoFF1-score
Authors
Shuai Li, Duc Manh Vu, Juergen Gall
Abstract
This paper concerns the problem of unsupervised temporal action segmentation for long, untrimmed videos. Recent successful approaches follow a joint representation learning and clustering paradigm, where optimal transport (OT) is adopted to produce pseudo labels for learning frame representations. These approaches alternate between estimating pseudo labels using OT and optimizing the parameters with gradient descent during training, where OT is used for obtaining the final temporal action segmentation. A major limitation of these works is that they learn a deterministic embedding for frame representations. The iterative procedure between learning deterministic embeddings based on pseudo labels and estimating pseudo labels from the learned embedding can thus get quickly stuck in a local optimum. As an alternative, we thus propose to learn a probabilistic embedding for frame representations. The embeddings are modeled by Gaussian distributions and we sample from the distributions before estimating the pseudo labels. We evaluate our approach on several challenging temporal action segmentation datasets and achieve results comparable to, and in some cases, better than the state of the art. Compared to baselines with deterministic embeddings, our approach improves MoF up to 20.7\% and F1-score up to 19.0\%. Our code is available at https://github.com/derkbreeze/PEOT.