HPC Data Provenance – Postdoctoral Researcher
Company Description
Join us and make YOUR mark on the World!
Lawrence Livermore National Laboratory (LLNL) has turned bold ideas into world-changing impact advancing science and technology to strengthen U.S. security and promote global stability.
Our mission spans four critical national security areas nuclear deterrence, threat preparedness, energy security, and multi-domain defense empowering teams to take on the toughest challenges of today and tomorrow. With a culture built on innovation and operational excellence, LLNL is a place where your expertise can make a real impact.
Job Description
We are seeking a Postdoctoral Researcher to perform research in data provenance, multi-layer tracing, and I/O characterization for large-scale scientific and AI workflows on high-performance computing (HPC) systems. Scientists increasingly need to answer questions such as which data produced a result, what transformed it, and which downstream models and analyses are affected when that data changes. Today the evidence needed to answer these questions is fragmented across hardware, operating system, I/O, runtime, and workflow layers. You will research and develop low-overhead techniques to automatically capture provenance from heterogeneous sources (e.g., application- and workflow-level tracers, data registries, and kernel-level instrumentation such as eBPF), assign stable identities to data artifacts as they move through workflows, and normalize and temporally align these records into a common representation that supports trustworthy, auditable lineage at HPC scale. You will work with domain scientists to validate these capabilities on leading-edge AI-for-science workflows running on Livermore Computing production systems. This position is in the Center for Applied Scientific Computing (CASC) within the Computing Directorate.
In this role you will
- Research and develop low-overhead, multi-layer provenance and I/O capture techniques for HPC workflows spanning the application, runtime, I/O library, file system, and operating system layers.
- Research and develop methods to integrate capture with HPC resource managers (e.g., Flux, Slurm) so that provenance is collected automatically without requiring changes to user workflows.
- Design and implement scalable registries that assign stable identities to datasets, model checkpoints, and workflow stages and reconcile the names used for them across different tools.
- Research and develop techniques to normalize, compress, and temporally align high-volume heterogeneous trace and event streams into a common schema.
- Characterize and minimize the performance overhead of tracing and provenance capture on large-scale deep learning and scientific workloads.
- Develop benchmarks, ground-truth datasets, and validation methodologies to evaluate provenance correctness, coverage, and overhead.
- Develop high-quality, open-source software that is extensible and maintainable by downstream users and developers.
- Deploy and evaluate software on Laboratory production HPC systems and interact with application scientists to u...
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
Sed ut perspiciatis unde omnis iste natus error sit voluptatem accusantium doloremque laudantium, totam rem aperiam, eaque ipsa quae ab illo inventore veritatis.