Neural architecture search made faster without training predictors

DIFFNAS: Predictor-Free Neural Architecture Search via Diffusion Model and Zero-Cost Evaluation

Machine Learning

Summary

Designing deep neural networks usually takes a lot of time because it needs training many models or using trained predictors to guess performance. The authors present DIFFNAS, a method that skips training predictors by using a diffusion model guided by quick-to-compute zero-cost performance estimates. This makes it much faster to find good network designs, achieving top accuracy on some image tasks while cutting search time substantially. However, in some complex design spaces, its accuracy is not as high as other methods that use predictors.

What this means in practice

  • For deep learning engineers: Speed up automated neural network design without costly model training or performance predictors using diffusion models guided by zero-cost metrics.
  • For mobile app developers: Use efficient architecture search to create smaller, faster neural networks suited for mobile devices with reduced computational resources.

Authors

Yize Sun, Hanzhang Ma, Zixin Wu, Xuechun Li, Volker Tresp, Yunpu Ma

Abstract

Neural Architecture Search (NAS) has advanced the automated design of deep networks, but remains computationally expensive due to reliance on model training or performance predictors. We introduce DIFFNAS, a predictor-free NAS framework that leverages a discrete graph diffusion model guided by zero-cost performance proxies. Unlike prior diffusion-based NAS methods that require learned performance predictors, DIFFNAS conditions the generative process on training-free metrics such as MeCo and SWAP, enabling efficient architecture search with minimal computational overhead. Experiments on NAS-Bench-201 show that DIFFNAS finds the optimal architectures for CIFAR-10 and CIFAR-100 (94.37% and 73.51%) without a pretrained predictor, while reducing search cost by up to 17.5x compared to DiffusionNAG. On the MobileNetV3 search space, its accuracy is lower than that of DiffusionNAG. These results highlight the promise of diffusion models combined with zero-cost proxies for scalable and effective architecture search.