Abstract
At the SDCC we are deploying a Jupyterhub infrastructure to enable scientists from multiple disciplines to access our diverse compute and storage resources. One major design goal was to avoid rolling out yet another compute backend, but rather to leverage our pre-existing resources via our batch systems (HTCondor and Slurm). Challenges faced include creating a frontend that allows users to choose what HPC resources they have access to as well as selecting containers or environments, delegating authentication to a MFA-enabled proxy, and automating deployment of multiple hub instances. This paper covers the design and features of our Jupyterhub service.
Highlights
The Scientific Data Center (SDCC) at Brookhaven National Laboratory serves an increasingly diverse community of more than two thousand users on over twenty projects ranging across the HEP, HENP, Photon Sciences, Astrophysics, LQCD and Condensed Matter disciplines
It provides a large High-Throughput Computing (HTC) farm accessed by the HTCondor batch system, along with experiment-specific job management layers, as well as a set of High-Performance Computing (HPC) clusters accessed via Slurm
The diversity of resources and access modes at the SDCC allows for a multiplicity of computing modes and workflows
Summary
The Scientific Data Center (SDCC) at Brookhaven National Laboratory serves an increasingly diverse community of more than two thousand users on over twenty projects ranging across the HEP, HENP, Photon Sciences, Astrophysics, LQCD and Condensed Matter disciplines It provides a large High-Throughput Computing (HTC) farm accessed by the HTCondor batch system, along with experiment-specific job management layers, as well as a set of High-Performance Computing (HPC) clusters accessed via Slurm. 2.2 Multi-factor Authentication The SDCC requires two-factor authentication for interactive access to its compute resources This requirement is fulfilled for the analysis portal by the front-end proxy interface to the Keycloak-based SDCC authentication infrastructure[2]. One of the most efficient use cases for Jupyter at the SDCC will be to act as a type of workflow management interface to the batch resources, replacing the typical array of shell scripts users employ to generate batch-job descriptions To this end, we developed a proof-of-concept library that inspired the HTMap Condor Library[4]. Dask integration is well-tested and fully functional with Slurm, and currently in beta stage with HTCondor, where it has been tested with simple workflows
Sign-in/Register to view highlights & summary 
Published Version (
Free)
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call