Mode Connectivity Beyond Classifiers: Evidence from Generative and Contrastive Models
2026-08-31 • Machine Learning
Machine Learning
AI summaryⓘ
The authors studied how different well-trained deep learning models relate to each other in terms of their training landscapes. Previous work showed that for image classifiers, you can smoothly connect two separate trained models without losing performance. The authors extended this idea to more complex types of models used for generating images (DDPM) and for learning image and text features together (NanoCLIP). They developed a new method tailored to these models' unique structures and found that these models also have smooth connections between them. This helps us better understand the shape and properties of the training landscapes in these advanced models.
Deep Neural NetworksLoss LandscapeMode ConnectivityDDPMNanoCLIPGenerative ModelsContrastive LearningNon-Convex OptimizationArchitecture-Aware Algorithms
Authors
Chengzheyi Yao, Yongzhao Zhang, Yongding Tian
Abstract
The loss landscape of Deep Neural Networks (DNNs) exhibits highly complex and non-convex properties. Recent studies have revealed the phenomenon of mode connectivity, demonstrating that independently trained network modes can be connected via a continuous low-loss path. However, existing mode connectivity research is predominantly confined to classifier-based models, leaving it an open question whether similar geometric properties exist in modern complex models. In this paper, we extend the boundaries of mode connectivity to generative and contrastive domains (specifically DDPM and NanoCLIP). Addressing the unique architecture of DDPM and CLIP, we propose an architecture-aware connection building algorithm. Extensive empirical results demonstrate for the first time that we successfully discover mode connectivity between independently trained DDPM and NanoCLIP modes. Our work provides a novel perspective for understanding the geometric properties of the loss landscapes in modern generative and contrastive models.