Framework verifies training process integrity for deep learning models
SciCheck: A Framework for Verifying the Integrity of Deep Learning Training
Cryptography and Security
Summary
Training deep learning models can be hard to verify because results vary every time and it's difficult to check if the process was done honestly. The authors created SciCheck, a method that records how training is done and lets a verifier check if the training matches expectations. It uses a two-step process: comparing statistics during training and re-running random segments to find any cheating or mistakes. This approach was tested on several model types and detected many problems while rarely rejecting honest efforts.
What this means in practice
- •For machine learning engineers: Confirm integrity of deep learning training runs by verifying recorded steps and randomly re-executing segments to detect tampering or errors.
- •For security teams in ai development: Detect and prevent attacks such as gradient sign flipping or label poisoning during model training with statistical and spot-check validation.
Authors
Alexander Nguyen, Dylan Gresham, Gaby G. Dagher, Min Long
Abstract
Deep learning training is difficult to verify within a scientific workflow: its architecture, hyperparameters, and procedure are rarely fixed until training has been attempted, and two honest runs never produce the same weights, so recomputation cannot serve as a check. We present SciCheck, a framework in which the scientist explores training configurations freely and posts a blueprint to a provenance ledger for each candidate they want verified, and a verifying scientist judges the resulting run in two layers: a statistical judge model that compares per-step gradient statistics against the verifier's own reference run, and a spot-check that re-executes randomly chosen training segments whose checkpoints are committed to the ledger one epoch at a time. On an MLP, a CNN, and an autoencoder at 100k-500k parameters, under gradient sign flipping, label poisoning, and fabricated runs, SciCheck reaches F1 of 0.966, 0.906, and 0.976 while rejecting 0-4.4% of honest runs. For the attacks we test, the spot-check accounts for most of this accuracy, and we characterize what each layer can detect. The effective-signal statistic falls as $(1-2s)^2$ under a sign flip of severity $s$, so gradient statistics alone catch strong enough sign flips, while fabricated runs that replay genuine statistics look no more anomalous than honest runs to any statistical detector we test. The spot-check catches fully fabricated segments with probability at least $1-(1-s)^k$ for $k$ sampled segments when the draws stay hidden until the run is committed, which roughly matches our measurements and lets a verifier choose $k$ for a target soundness. Verification costs 1.5-2.3x one retraining, mostly for the verifier's reference run; each submission's spot-check takes under 1.5 s, and per-epoch commitments reduce checkpoint storage 5-17x compared with keeping every checkpoint.