Experiences with Cross-Facility Real-Time Light Source Data Analysis Workflows

Abstract

We are seeing a growth in scientific data from experimental and observational facilities that are resulting in significant new computational patterns and needs. For example, scientists running experiments at light sources, often analyses workflows require near real-time access to compute resources in order to obtain results used for re-configuring on-going experiments. These workflows often have requirements that are different from the traditional large-scale parallel applications that have traditionally run at HPC centers. In this paper, we present our experiences supporting two light source data analysis workflows that run on HPC resources at National Energy Research Scientific Computing Center. We discuss the characteristics of workflows, runtime requirements and associated execution challenges when running on HPC environments. We present a discussion and a summary of best practices that address execution challenges and current and future solutions for leveraging HPC resources for near real-time data analysis.