Stateful testing exposes hidden bugs in deep learning compilers
Stateful Testing for Deep Learning Compilers
Software EngineeringMachine Learning
Summary
Deep learning compilers often keep some information between runs, which can cause errors that don't show up when you run things just once. The authors studied many such bugs in popular frameworks like PyTorch and TensorFlow and built a new tool called Rondo to find them. Rondo tests sequences of compiler runs that first set up the compiler state and then try to trigger a bug. It uses smart mutations and comparisons to find bugs related to compiler state and can shorten the test cases to make debugging easier. This method found and helped fix many real bugs in widely used systems.
What this means in practice
- •For machine learning engineers: Detect and diagnose subtle state-related bugs when compiling deep learning models to improve reliability.
- •For software testers: Generate and shrink test sequences that expose compiler state bugs in complex model compilation workflows.
Authors
Jiatao Ge, Ruixiang Qian, Yibo Hu, Yanzhou Mu, Yinglong Zou, Jiawei Liu, Zhenyu Chen, Chunrong Fang
Abstract
Modern deep learning compilers compile models lazily and cache artifacts across invocations. Retained compiler state can make a later invocation incorrect even when it is correct in isolation. An analysis of 26 publicly reported issues in PyTorch, JAX, and TensorFlow shows that detecting such bugs requires sequences that first establish compiler state and then expose a failure. We present Rondo, a stateful testing method that represents each test case as a model paired with an invocation sequence. Rondo mutates invocation sequences and uses a Markov chain Monte Carlo sampler guided by compilation behavior to favor rarely observed behaviors. Its three-way differential oracle compares execution without compilation, compiled execution with compiler state reset, and compiled execution with state retained. It confirms stateful bugs by replaying the same invocation with state retained and reset while preserving inputs and program state, then shortens sequences while preserving the original bug. We evaluate Rondo on four deep learning frameworks using 24 common model architectures. Rondo detects 117 bugs, including 35 stateful bugs. We submit 39 bug reports, including 18 for stateful bugs; developers confirm 35 reports, including 16 for stateful bugs, and fix 18 bugs, including 11 stateful bugs. Rondo achieves the highest behavior coverage and compiler Python source-line coverage among the compared methods, with 99.26% alarm precision. It shortens 44 of 117 invocation sequences, reducing the lengths of these 44 sequences by 83.34% on average.