Self supervised pretraining improves particle identification with large labeled data
SuperJEPA: Pretraining with a Supervision-Augmented Joint-Embedding Predictive Architecture for High Energy Physics
Machine Learning
Summary
High energy physics experiments generate lots of labeled simulated data to identify particles. The authors explore whether adding self-supervised learning methods helps improve particle identification beyond just using labeled data alone. They introduce a new training method called SuperJEPA that combines self-supervised and supervised learning. Their results show that this combined approach improves how well particle types can be identified, even when large labeled datasets are available.
What this means in practice
- •For particle physics software developers: Improve particle type classification software using combined self-supervised and supervised training to boost accuracy in jet tagging tasks.
- •For machine learning engineers: Enhance model training pipelines for tasks with abundant labeled simulated data by integrating predictive self-supervision with supervised objectives.
Authors
Dylan Sheldon Rankin, Ho Fung Tsoi
Abstract
Foundation models often obtain powerful results through very large unlabeled training datasets that dwarf available labeled datasets in size, but high energy physics already provides large samples of high-fidelity labeled simulation. We therefore ask whether self-supervised objectives improve over supervised pretraining alone when large labeled samples already exist. We introduce SuperJEPA, a joint-embedding predictive architec- ture augmented with a supervised objective, and find improved downstream jet-tagging performance across multiple tasks, model sizes, and pretraining dataset sizes. The gains persist for both frozen and fine-tuned encoders, motivating predictive self-supervision in future large-scale particle-physics foundation models.