Data management and processing workflow for the Materials Physics and Engineering group beamlines at the Advanced Photon Source.

Jun-Sang Park,Siniša Veseli,Peter Kenesei,Prithvi Ramanathan,Connor Horn

doi:10.1107/s1600577519000584

Abstract

The ability to store, organize, process and distribute experimental data effectively, efficiently and securely is particularly important for large user facilities like the Advanced Photon Source. In this article, the deployment of the APS Data Management System (DM) at the 1-ID and 6-BM beamlines of the APS is described. These two beamlines support a wide range of experimental techniques and generate data at relatively high rates, making them ideal candidates to illustrate the deployment and customization of the DM system and its tools. Using several usage examples at these beamlines, various capabilities of the DM system are described.

Highlights

The ability to store, organize, process and distribute experimental data effectively, efficiently and securely is important as user facilities, like the Advanced Photon Source (APS), host a growing number of users, develop more complex suites of experimental techniques, and generate larger volumes of experimental data
Details about the Data Management System (DM) system architecture are presented elsewhere (Veseli et al, 2018); here, we focus on the deployment and customization of the DM system for the 1-ID and 6-BM beamlines of the APS where a wide range of experimental techniques using high-energy X-rays are hosted
In x3, we provide an overview of the experimental techniques supported by the APS 1-ID and 6-BM beamlines and the data generation rates associated with these techniques

Summary

Introduction

The ability to store, organize, process and distribute experimental data effectively, efficiently and securely is important as user facilities, like the Advanced Photon Source (APS), host a growing number of users, develop more complex suites of experimental techniques, and generate larger volumes of experimental data. The National Synchrotron Light SourceII (NSLS-II) have deployed a comprehensive data acquisition and management system (http://nsls-ii.github.io/index.html) (Arkilic et al, 2017) These solutions are often augmented by data processing or analysis workflow management systems. These solutions have similar goals but are customized and developed to address the unique needs of each facility. The DM system (Veseli et al, 2018) developed at the APS offers a common framework and set of tools around which the beamlines can build their data acquisition, storage, distribution and processing workflows. When a new file is transferred into the DM Storage the storage management service ensures that the file is associated with the

Data management system architecture

DM deployment at the APS 1-ID and 6-BM beamlines

Data management workflow at 1-ID and 6-BM beamlines

Findings

Summary and future work