Trace scientific data origins inside distributed workflows automatically
TRACEs: Fine-Grained Embedded Provenance for Distributed Scientific Workflows
Distributed, Parallel, and Cluster ComputingComputational Engineering, Finance, and Science
Summary
It can be hard to understand how scientific results come about when many computers across different locations work together, especially since similar runs can give different answers. The authors present TRACEs, a system that stores detailed information about how every piece of data was created, directly inside the data itself. This approach works even when workflows run on many computers and doesn’t disrupt current management tools. They showed TRACEs can fully track data origins in a complex biology workflow across computers, using less storage and without slowing the process much, plus it can explain why results vary without rerunning experiments.
What this means in practice
- •For hpc operations teams: Track detailed data lineage embedded within computational outputs across distributed high-performance computing resources to improve debugging and reproducibility.
- •For bioinformatics engineers: Diagnose causes of differing genetic variant results in distributed genome analysis pipelines by querying embedded provenance without rerunning workflows.
Authors
Ty Anderson, Kin Ng, Rajiv Mayani, Jay Lofstead, Ewa Deelman, Michela Taufer
Abstract
Large-scale scientific workflows run on distributed HPC resources, making it hard to explain how scientific data products are generated and why similar workflow executions can yield very different results. Existing systems usually focus on either distributed workflow execution or provenance management and rarely treat fine-grained provenance as an intrinsic property of distributed scientific data artifacts. We present TRACEs (Traceable Runtime for Advanced Containerized Executions), a provenance architecture for distributed containerized scientific workflows that embeds data lineage and execution context directly in every generated data artifact. By decoupling provenance from workflow orchestration, TRACEs generalizes artifact-centric provenance from local container runs to distributed workflows while remaining compatible with existing workflow management systems. We evaluate TRACEs on a seven-stage E. coli variant-calling workflow distributed across multiple HPC nodes. Our results show 100% provenance completeness under distributed execution, about 59% less intermediate storage via sparse IMG containers, and runtime and zero-copy I/O throughput comparable to fixed-size SIF containers, plus provenance-guided explanations of workflow variability that trace a 200-fold difference in variant counts to differences in input sequencing data without re-executing the workflow.