Insights into Research Computing Operations using Big Data-Powered Log Analysis

Abstract

Research computing centers provide a wide variety of services including large-scale computing resources, data storage, high-speed interconnect and scientific software repositories to facilitate continuous competitive research. Efficient management of these complex resources and services, as well as ensuring their fair use by a large number of researchers from different scientific domains are key to a center's success. Almost all research centers use monitoring services based on real time data gathered from systems and services, but often lack tools to perform a deeper analysis on large volumes of historical logs for identifying insightful trends from recurring events. The size of collected data can be massive, posing significant challenges for the use of conventional tools for this kind of analysis. This paper describes a big data pipeline based on Hadoop and Spark technologies, developed in close collaboration between TACC and Georgia Tech. This data pipeline is capable of processing large volumes of data collected from schedulers using PBSTools, making it possible to run a deep analysis in minutes as opposed to hours with conventional tools. Our component-based pipeline design adds the flexibility of plugging in different components, as well as promotes data reuse. Using this data pipeline, we demonstrate the process of formulating several critical operational questions around researcher behavior, systems health, operational aspects and software usage trends, all of which are critical factors in determining solutions and strategies for efficient management of research computing centers.