Procedural input search improves neural network robustness on vision tasks
Procedural Adversarial Learning
Computer Vision and Pattern Recognition
Summary
Neural networks trained for vision tasks sometimes make mistakes on simple scenes that aren’t in their training data. The authors propose a method called Procedural Adversarial Learning (PAL) that uses a program to generate images with adjustable settings, like lighting or pose, to find hard examples that cause errors. They created an algorithm called Simple-PAT that searches these settings to find tough examples and uses them to train networks more robustly. Their method significantly reduces worst-case errors compared to standard and other robust training approaches.
What this means in practice
- •For computer vision engineers: Train vision models to handle challenging, rare cases by generating tough training images on demand.
- •For robotics developers: Improve perception systems by exposing them to difficult synthetic scenes created through parameterized procedural inputs.
Authors
Jack Nugent, Jia Deng
Abstract
For many computer vision tasks, neural networks can be trained to produce a low average error, yet they still fail on seemingly simple scenes. Reducing these failures requires first finding them, but they may not be in a fixed dataset. We study procedural adversarial learning (PAL), a framework of minimax training and adversarial evaluation which searches the input space of a procedural generator for difficult instances. The procedural generator is an interpretable program mapping a low-dimensional vector of control parameters (e.g. layout, lighting, pose) to a rendered image and its label. We introduce the Simple-PAT algorithm, which uses zeroth-order optimization in the procedural generator's parameter space to propose adversarial training samples that are generated on demand. We also create PAL-Dojo, a collection of efficient procedural generators for surface normal estimation, image segmentation, and human pose estimation. Against a cross-model black-box adversary with 50M aggregate queries, Simple-PAT reduces worst-case error by a geometric mean of 3.15x compared to standard training and 2.63x compared to the strongest robust training baseline. Code is available at https://github.com/princeton-vl/procedural-adversarial-learning.