Training vision models improves with better choice of Inception crop scales
You May Be Running the Wrong Inception Crop
Machine Learning
Summary
The paper reveals that a commonly used method for preparing images during training is often implemented differently across popular software, which can affect model performance. The authors trained many vision transformer models on a large image dataset and found that the exact way image crops are sampled matters, especially for longer training. They show that stronger and more flexible cropping strategies help models learn better, leading to improved accuracy and clearer model focus. Their findings hold true across different optimization methods, suggesting broad relevance.
What this means in practice
- •For deep learning engineers: Optimize training of vision transformer models by adjusting crop scale distributions based on training duration and augmentation strength.
- •For machine learning platform developers: Improve image augmentation modules by implementing correct crop scale sampling to enhance model performance consistency across frameworks.
Authors
Jason Chuan-Chih Chou
Abstract
A decade after its inception, Inception crop has become the standard crop-based data augmentation method for training deep vision models. Not only is its practice of uniformly sampling crop scale and aspect ratio widely adopted, but also its lower and upper bounds, with the scale lower bound being the sole exception that is sometimes tuned. It is therefore surprising that the standard implementation in the TensorFlow / JAX ecosystem samples crop scale with probability density function $f(A) \propto \frac{1}{\sqrt{A}}$ unlike the PyTorch counterpart, which follows the original description. Motivated by this discovery, we train 522 ViT-S/16 models on the ImageNet-1k dataset with various training budgets and crop scale distributions. We reach $78.78\pm0.09$ top-1 val. accuracy with 90 epochs of training budget and find that 1. Higher training budget requires stronger augmentation; 2. Lower tail of the distribution of the crop scale determines the augmentation strength of Inception crop; 3. Models trained with higher training budget exhibit sparser saliency, regardless of the crop scale distribution or weight decay. Based on 2. we revisit the performance of Beta crop, whose softer cutoff allows it to optimize model performance across training budgets with less compromise. We replicate 1. and 3. with Scion optimizer in addition to AdamW, suggesting that the results may be general.