Teacher-guided method speeds neural network search for tinyml devices
Rank-Reliable Teacher-Guided Fitness Approximation for Expensive Evolutionary Optimization: A TinyML Architecture Search Study
Artificial IntelligenceMachine Learning
Summary
Finding the best small neural networks for devices like smart sensors can take a long time because each possibility needs costly testing. This paper presents a way to quickly guess which networks are better by learning from a teacher model that ranks candidates by difficulty and class. Their approach combines this quick guess with a statistical model to pick promising candidates while reducing the number of expensive tests. Tests on audio recognition tasks show their method is faster and more reliable than full evaluation methods.
What this means in practice
- •For embedded system developers: Accelerate the design of energy-efficient neural networks for devices with limited compute by quickly ranking promising architectures before full testing.
- •For mobile app developers: Improve on-device AI performance by efficiently searching for neural network designs that meet constrained budgets using teacher-guided evaluation.
Authors
Soumen Garai, Suman Samui
Abstract
Expensive evolutionary search does not always need an exact fitness estimate for every candidate. It often needs a reliable answer to a simpler question: which candidate is better? We address this need through Teacher-Guided Learning NSGA-II (TGL-NSGA-II), a low-fidelity framework for constrained Tiny Machine Learning (TinyML) neural architecture search. A pretrained teacher organizes samples into strata defined jointly by difficulty and class. Each candidate then undergoes KD-Lite, a short and capped knowledge-distillation procedure on a compact training set, before being scored on a separate stratified evaluation set. This teacher-guided score is fused with a Gaussian-process surrogate to select candidates for full evaluation. For a fixed candidate population, we analyse evaluation variance, score concentration, pairwise rank inversion, expected Kendall-$τ$, first-front identification, and hypervolume perturbation. We also derive a variance-aware fusion weight and a capacity-adaptive distillation rule. On keyword spotting and bird-call classification, the measured Kendall-$τ$ values are 0.74 and 0.62, exceeding the corresponding predicted lower bounds of 0.60 and 0.46. Joint stratification reduces proxy-score variance by 41% relative to random evaluation. Selective teacher mismatch, in contrast, increases differential bias and reduces Kendall-$τ$ to 0.41. Under a constrained evaluation budget, TGL-NSGA-II achieves the largest mean hypervolume and smallest generational distance on keyword spotting, records the lowest mean false-positive rate on BirdCLEF, and runs 2.2x faster than full NSGA-II. These guarantees apply to population-level low-fidelity evaluation and do not establish convergence of the complete evolutionary trajectory.