Generation provenance improves auditing of synthetic speech data
Generation Provenance Before Behavior Attribution: Auditing Synthetic Speech Research Objects
Artificial IntelligenceComputation and LanguageMachine Learning
Summary
It can be hard to figure out which parts of synthetic speech training data caused certain behaviors in AI models. The authors introduce a way to keep detailed records showing how each synthetic audio clip was created and reviewed. They tested this method with synthetic Japanese speech in healthcare scenarios, finding it helps set up clear links between data and outcomes. However, knowing the origin alone is not enough to prove exactly how training data affected the model's behavior.
What this means in practice
- •For speech data teams: Track and verify the source and review history of synthetic speech clips to support compliance and auditing in voice AI projects.
- •For healthcare technology teams: Use detailed generation provenance to audit synthetic speech used in care handoff systems for improved data transparency and safety checks.
Tested on one dataset.
Authors
Sidi Chang, Peiying Zhu
Abstract
Attributing model behavior to synthetic training data requires knowing what produced each training item before estimating what that item caused. A waveform-label pair does not preserve this knowledge. We propose a generation-provenance substrate in which a synthetic research object binds source specification, generated content, waveform, target, fact requirements, quality signals, review lineage, and immutable manifest identity. Producer and selection mechanism determine evidentiary meaning; storage location and variable name do not. We audit this substrate in a private Japanese care-handoff pipeline. A 113-asset review population contains 1.552 hours of synthetic speech across six scenario families; all items have linked audio, transcripts, candidate notes, and fact checklists, but human evidence is selective and source-specific. Two faithful-only manifests are scenario-seed-disjoint and immutably versioned, while exact upstream attribution remains blocked by floating generator aliases, missing per-clip TTS and code stamps, and an unversioned checking prompt. We argue that generation provenance is necessary but not sufficient for behavior attribution: it defines the candidate causal graph and audit units, whereas contributive attribution still requires frozen training runs and intervention or influence evidence. The paper contributes a compact provenance contract, an audit protocol, and a bounded case study for synthetic-data attribution; controlled research access may be offered, but we do not claim causal training-data attribution, clinical validity, or unrestricted public release.