Vst-bench-ml ensures trustworthy audio plugin dataset creation and verification
vst-bench-ml: Auditable Dataset Generation for Audio Plugins
Sound
Summary
Creating datasets with audio effects plugins can be tricky because it's important to keep track of exactly how each sound was made. The authors created a tool called vst-bench-ml that carefully records all the settings and outputs when making sounds, so you can check later if everything matches up exactly. It uses a locked manifest file to prove that the recorded data hasn't been changed, and this checking works even without opening the plugin again. They tested it on a specific Mac setup and showed it can reproduce the results reliably, but they note it doesn’t guarantee the quality or usefulness of the sounds themselves.
What this means in practice
- •For audio software developers: Generate reliable, verifiable datasets of plugin parameter configurations and outputs for testing or training purposes.
- •For quality assurance teams: Ensure audio plugin outputs are reproducible and untampered during software development cycles.
Authors
Youssef Hemimy
Abstract
Generating datasets with audio plugins requires preserving how each audio observation was produced. vst-bench-ml is a Python package that samples plugin parameters, coordinates rendering through a separately installed host, and records software identities, rendering settings, and output hashes for later verification. A sealed manifest binds this evidence and supports offline integrity checks without loading the plugin. A companion repository provides a sample-free VST3 test plugin, a deterministic offline host, and additional checks linking rendering attempts to recorded outputs. On a pinned macOS arm64 and Python 3.12 stack, functional validation of the pre-release versions reproduced package and host tests. Two runs in fresh processes reproduced the same render-plan hash and all eight expected output hashes, covering four sampled parameter configurations and two fixed MIDI signals; 12 negative cases also passed. Later licensed 0.1.0 release candidates passed producer-run continuous integration and local validation. Version 0.1.1 adds public Git source repositories, package-index installation, and user documentation; a separate release check reproduced the expected plan and output hashes. These checks establish integrity and repeatability within the tested setup; they do not establish portability, scientific suitability, or perceptual quality.