Papers for

hpc operations teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Trace scientific data origins inside distributed workflows automatically

TRACEs: Fine-Grained Embedded Provenance for Distributed Scientific Workflows

Abstract: Large-scale scientific workflows run on distributed HPC resources, making it hard to explain how scientific data products are generated and why similar workflow executions can yield very different results. Existing systems usually focus on either distributed workflow execution or provenance management and rarely treat fine-grained provenance as an intrinsic property of distributed scientific data artifacts. We present TRACEs (Traceable Runtime for Advanced Containerized Executions), a provenance architecture for distributed containerized scientific workflows that embeds data lineage and execution context directly in every generated data artifact. By decoupling provenance from workflow orchestration, TRACEs generalizes artifact-centric provenance from local container runs to distributed workflows while remaining compatible with existing workflow management systems. We evaluate TRACEs on a seven-stage E. coli variant-calling workflow distributed across multiple HPC nodes. Our results show 100% provenance completeness under distributed execution, about 59% less intermediate storage via sparse IMG containers, and runtime and zero-copy I/O throughput comparable to fixed-size SIF containers, plus provenance-guided explanations of workflow variability that trace a 200-fold difference in variant counts to differences in input sequencing data without re-executing the workflow.

Thu 29 OctDistributed, Parallel, and Cluster ComputingComputational Engineering, Finance, and Science
The gist
It can be hard to understand how scientific results come about when many computers across different locations work together, especially since similar runs can give different answers. The authors present TRACEs, a system that stores detailed information about how every piece of data was created, directly inside the data itself. This approach works even when workflows run on many computers and doesn’t disrupt current management tools. They showed TRACEs can fully track data origins in a complex biology workflow across computers, using less storage and without slowing the process much, plus it can explain why results vary without rerunning experiments.
Open → 2610.39981v1