Blockwise scoring limits accuracy in neural architecture search

When Does Blockwise Additivity Break? Diagnosing and Repairing the Additive Assumption in Distillation-Based Architecture Search

Networking and Internet ArchitectureNeural and Evolutionary Computing

Summary

When designing smaller, faster neural networks by swapping building blocks, it's common to score each block separately and add those scores to decide the best combination. The authors showed that this 'additive' assumption works well for mild size reductions but breaks down when networks get smaller, causing big accuracy losses. They also separated why this happens and proposed a hybrid approach that selectively checks combinations, improving results without much extra work. Their findings hold true across different models and datasets.

What this means in practice

  • For deep learning engineers: Improve neural network compression by using hybrid block scoring that better captures block interactions while keeping search efficient.
  • For mobile app developers: Build faster, smaller models for devices by adopting architecture search methods that avoid additive scoring errors at high compression.

Authors

Yuefeng Xu, Jun Yu

Abstract

Blockwise distillation methods for architecture search, such as DNA, DONNA, LANA and Puzzle, score each candidate replacement block in isolation and assemble a child network by minimizing the sum of per-block scores. We test this additive assumption on DeiT-S and DeiT-B with seven replacement blocks per layer, seven block-latency budgets from 70% to 40%, three fine-tuning seeds and paired bootstrap confidence intervals. (i) At mild compression the additive integer linear program (ILP) is close to a pairwise integer quadratic program (IQP) that models block interactions, but the post-fine-tuning gap widens sharply between the 60% and 55% budgets: at 55%-40% the ILP trails the IQP by 3.3-16.6 top-1 points on ImageNet-100. (ii) We split the KL additivity error into a metric component, caused by the non-linearity of the KL in summed logit changes and computable from cached single-block logits, and a functional component, the score-level effect of non-additive logit changes. The metric component accounts for most of the error on DeiT-S (59-65%), the functional component for most of it on DeiT-B (73-76%). (iii) A hybrid search ranks moves with the logit-sum predictor and measures only the best ones. It needs no dedicated pair measurements, costs 13-27% of the IQP's pair stage, and on ImageNet-100 significantly outperforms the IQP in 7 of 14 settings, with no significant difference in 6 and a significant loss in 1. Further reducing the pre-fine-tuning KL, or scoring after a short fine-tune, gives no consistent gain. On full ImageNet-1k with 1000-class fine-tuning, the sharp widening between 60% and 55% (5.8-17.5 points at 55%-40%), the model-dependent decomposition and the hybrid's advantage (significantly better than the IQP in 13 of 14 settings, no the other) are reproduced.