HPC Clusters Research Articles

Analysis of workflow schedulers in simulated distributed environments

Task graphs provide a simple way to describe scientific workflows (sets of tasks with dependencies) that can be executed on both HPC clusters and in the cloud. An important aspect of executing such graphs is the used scheduling algorithm. Many scheduling heuristics have been proposed in existing works; nevertheless, they are often tested in oversimplified environments. We provide an extensible simulation environment designed for prototyping and benchmarking task schedulers, which contains implementations of various scheduling algorithms and is open-sourced, in order to be fully reproducible. We use this environment to perform a comprehensive analysis of workflow scheduling algorithms with a focus on quantifying the effect of scheduling challenges that have so far been mostly neglected, such as delays between scheduler invocations or partially unknown task durations. Our results indicate that network models used by many previous works might produce results that are off by an order of magnitude in comparison to a more realistic model. Additionally, we show that certain implementation details of scheduling algorithms which are often neglected can have a large effect on the scheduler’s performance, and they should thus be described in great detail to enable proper evaluation.

Benchmark study of 2D and 3D VOF simulations of a simplex nozzle using a hybrid RANS-LES approach

In this study, a simplex nozzle is tested with water for the benchmarking of different flow simulation models. A large scale Plexi-glass transparent nozzle is used to reduce the influence of production tolerances on the performance. Experiments are conducted at different flow rates and CD, spray angle and film thickness parameters are evaluated. 2D and 3D hybrid RANS-LES multiphase flow simulations of simplex nozzle are validated against the experimental data. Multiphase nature of the flow is modelled by volume of fluid method. The main goal is to assess the capabilities and drawbacks of 2D axisymmetric and full sector 3D modeling approaches. It is observed that although full sector 3D simulations require HPC cluster systems, accuracies in validation parameters are quite satisfying. Conversely, 2D axisymmetric simulations which can be run on a single core and give a general outlook of the flow field, they show an overshoot of CD and film thickness over the selected range of flow rate. It is shown that this overshoot is mostly related with the inlet boundary condition, which can not take the flow contraction and/or separation at the inlet slots into account. After correcting the inlet velocity 2D simulations by using the 3D results, it is shown that the predictions can be quite close to the experimental data.

Toward highly parallel loading of unstructured meshes

This paper presents an algorithm for highly-parallel loading and processing of unstructured mesh databases in a distributed memory environment of large HPC clusters without collecting data into a single process. The algorithm is proved effective, having linear speedup in the large dataset limit. Demonstrated on Ansys CDB, EnSight, VTK Legacy, and XDMF databases, we show that it is possible to efficiently reconstruct meshes with 800 million nodes and 500 million elements in several seconds on thousands of processors, even from databases that were not designed to be read in parallel. The algorithm is implemented in our MESIO library that can be used as (i) an efficient parallel loader (e.g. for numerical physical solvers) or as (ii) a high performing parallel converter between mesh databases.

A Head Loss Pressure Boundary Condition for Hydraulic Systems

Despite the increase in computational power of HPC clusters, it is in most cases not possible to include the entire hydraulic system when doing detailed numerical studies of the flow in one of the components in the system. The numerical models are still most often constrained to a small part of the system and the boundary conditions may in many cases be difficult to specify. The headLossPressure boundary condition is developed in the present work for the OpenFOAM open-source CFD code to include the main effects caused by a large hydraulic system onto a component in the system. The main motivation is to provide a boundary condition for hydraulic systems where known properties are specified by the user and unknown properties are calculated. This paper is a guide to the developed headLossPressure boundary condition. It is based on the extended Bernoulli equation to calculate the kinematic pressure on the patch. An arbitrary number of minor and friction losses are considered to describe the system in terms of head losses. The boundary condition also provides the opportunity to specify the head in relation to a reference elevation. System changes during operations are modelled through Function1 variables, which enables time-varying inputs. The developments are validated against experimental test data, where the varying head between two free surfaces and a valve closing and opening sequence are modelled with the boundary condition. The main effects of the system are well captured by the headLossPressure boundary condition. It is thus a useful and trustworthy boundary condition for incompressible hydraulic system simulations.

Quantifying the health effects of exposure to non-exhaust road emissions using agent-based modelling (ABM)

This paper provides an agent-based model, entitled TRAPSim, to examine the exposure to non-exhaust emissions (NEEs) and the consequent health effects of driver and pedestrians groups in Seoul. To make the model reproducible and replicable, TRAPSim uses the ODD protocol to demonstrate the details of the agents and parameters, as well as provide the codes alongside the descriptions to avoid possible ambiguity. The model’s main parameters are thoroughly tested through sensitivity experiments and are calibrated with the city’s air pollution monitoring networks. This paper also provides the instructions to the model, possible artefacts, and the configurations to submit the model on the HPC cluster.•An ODD protocol is used to document the agent-based model TRAPSim.•Sensitivity experiments and calibration are explained.•The step-by-step codes and annotations are attached in the protocol and HPC sections.

Design considerations for workflow management systems use in production genomics research and the clinic

The changing landscape of genomics research and clinical practice has created a need for computational pipelines capable of efficiently orchestrating complex analysis stages while handling large volumes of data across heterogeneous computational environments. Workflow Management Systems (WfMSs) are the software components employed to fill this gap. This work provides an approach and systematic evaluation of key features of popular bioinformatics WfMSs in use today: Nextflow, CWL, and WDL and some of their executors, along with Swift/T, a workflow manager commonly used in high-scale physics applications. We employed two use cases: a variant-calling genomic pipeline and a scalability-testing framework, where both were run locally, on an HPC cluster, and in the cloud. This allowed for evaluation of those four WfMSs in terms of language expressiveness, modularity, scalability, robustness, reproducibility, interoperability, ease of development, along with adoption and usage in research labs and healthcare settings. This article is trying to answer, which WfMS should be chosen for a given bioinformatics application regardless of analysis type?. The choice of a given WfMS is a function of both its intrinsic language and engine features. Within bioinformatics, where analysts are a mix of dry and wet lab scientists, the choice is also governed by collaborations and adoption within large consortia and technical support provided by the WfMS team/community. As the community and its needs continue to evolve along with computational infrastructure, WfMSs will also evolve, especially those with permissive licenses that allow commercial use. In much the same way as the dataflow paradigm and containerization are now well understood to be very useful in bioinformatics applications, we will continue to see innovations of tools and utilities for other purposes, like big data technologies, interoperability, and provenance.

JUSUF: Modular Tier-2 Supercomputing and Cloud Infrastructure at Jülich Supercomputing Centre

JUSUF is a petaflop supercomputer operated by Jülich Supercomputing Centre at Forschungszentrum Jülich as a European supercomputing and cloud resource. JUSUF was funded via the ICEI project and especially serves the Human Brain Project and PRACE via ICEI and the Fenix Research Infrastructure. The system consists of two parts, an HPC cluster partition and an Infrastructure-as-a-Service cloud partition. The system entered production phase in spring 2020. It is based on the Bull X400 product family with AMD Rome processors, partially accelerated by Nvidia V100 GPUs, and Nvidia Mellanox HDR InfiniBand.

Generating UAV high-resolution topographic data within a FOSS photogrammetric workflow using high-performance computing clusters

Photogrammetry is one of the most reliable techniques to generate high-resolution topographic data and it is key to territorial mapping and change detection analysis of landforms in hydro-geomorphological high-risk areas. Specifically, the Structure from Motion (SfM) is an emerging topographic survey technique that addresses the problem of determining the 3D position of image descriptors to estimate three-dimensional structures. Thanks to the potential of SfM algorithm and the development of Unmanned Aerial Vehicles (UAVs) that allow the on-demand acquisition of high-resolution aerial images, it is possible to survey extended areas of the Earth surface and monitor active phenomena through multi-temporal surveys. However, the ability to detect remote and wide areas with a very high-resolution is countered by the need to capture large datasets which can limit the photogrammetric process, due to the need for high-performance hardware. This paper presents a photogrammetric workflow based on Free and Open-Source Software (FOSS), which is able to return different outputs and to manage a large amount of data in reasonable time, through the distribution of the most computationally expensive steps on computing clusters hosted by the ReCaS-Bari data center for scientific research. The results are given in terms of performance evaluations based on different computing configurations of the clusters and setups of the steps of the workflow. The HTC cluster test with a parallel SSH approach involved an important reduction of several hours in the processing time of thousands UAV images, especially compared to classic photogrammetric process on a single workstation with commercial software.A parallel test, aimed to validate the performance of a single sever of the new HPC cluster, involved really good results halving the processing time with respect to the HTC cluster test.

Momentum-driven adaptive synchronization model for distributed DNN training on HPC clusters

Building a distributed deep learning (DDL) system on HPC clusters that guarantees convergence speed and scalability for the training of DNNs is challenging. The HPC cluster, which consists of multiple high-density multi-GPU servers connected by the Infiniband network (HDGib), compresses the computing and communication time for distributed DNNs' training but brings new challenges. The convergence time is far from linear scalability (with respect to the number of workers) for parallel DNNs training. We thus analyze the optimization process and identify three key issues that cause scalability degradation. First, the high-frequency update for parameters due to the compression of the computing and communication times exacerbates the stale gradient problem, which slows down the convergence. Second, the previous methods used to constrain the gradient noise (stochastic error) of the SGD are outdated, as HDGib clusters can support more strict constraints due to the Infiniband network connections, which can further constrain the stochastic error. Third, the same learning rate for all workers is inefficient due to the different training stages of each worker. We thus propose a momentum-driven adaptive synchronization model that focuses on solving the above issues and accelerating the training procedure on HDGib clusters. Our adaptive k-synchronization algorithm uses the momentum term to absorb the stale gradients and adaptively bind the stochastic error to provide an approximate optimal descent direction for the distributed SGD. Our model also includes an individual dynamic learning rate search method for each worker to further improve training performance. Compared with previous linear and exponent decay methods, it can provide a more precise descent distance for distributed SGD based on different training stages. Extensive experimental results indicate that the proposed model effectively improves the training performance of CNNs, which retains high accuracy with a speed-up of up to 57.76% and 125.3% on the CPU-based and GPU-based clusters, respectively.

Workflow provenance in the lifecycle of scientific machine learning

AbstractMachine learning (ML) has already fundamentally changed several businesses. More recently, it has also been profoundly impacting the computational science and engineering domains, like geoscience, climate science, and health science. In these domains, users need to perform comprehensive data analyses combining scientific data and ML models to provide for critical requirements, such as reproducibility, model explainability, and experiment data understanding. However, scientific ML is multidisciplinary, heterogeneous, and affected by the physical constraints of the domain, making such analyses even more challenging. In this work, we leverage workflow provenance techniques to build a holistic view to support the lifecycle of scientific ML. We contribute with (i) characterization of the lifecycle and taxonomy for data analyses; (ii) design decisions to build this view, with a W3C PROV compliant data representation and a reference system architecture; and (iii) lessons learned after an evaluation in an Oil & Gas case using an HPC cluster with 393 nodes and 946 GPUs. The experiments show that the decisions enable queries that integrate domain semantics with ML models while keeping low overhead (<1%), high scalability, and an order of magnitude of query acceleration under certain workloads against without our representation.

Ensemble Prediction of Job Resources to Improve System Performance for Slurm-Based HPC Systems.

In this paper, we present a novel methodology for predicting job resources (memory and time) for submitted jobs on HPC systems. Our methodology based on historical jobs data (saccount data) provided from the Slurm workload manager using supervised machine learning. This Machine Learning (ML) prediction model is effective and useful for both HPC administrators and HPC users. Moreover, our ML model increases the efficiency and utilization for HPC systems, thus reduce power consumption as well. Our model involves using Several supervised machine learning discriminative models from the scikit-learn machine learning library and LightGBM applied on historical data from Slurm.Our model helps HPC users to determine the required amount of resources for their submitted jobs and make it easier for them to use HPC resources efficiently. This work provides the second step towards implementing our general open source tool towards HPC service providers. For this work, our Machine learning model has been implemented and tested using two HPC providers, an XSEDE service provider (University of Colorado-Boulder (RMACC Summit) and Kansas State University (Beocat)).We used more than two hundred thousand jobs: one-hundred thousand jobs from SUMMIT and one-hundred thousand jobs from Beocat, to model and assess our ML model performance. In particular we measured the improvement of running time, turnaround time, average waiting time for the submitted jobs; and measured utilization of the HPC clusters.Our model achieved up to 86% accuracy in predicting the amount of time and the amount of memory for both SUMMIT and Beocat HPC resources. Our results show that our model helps dramatically reduce computational average waiting time (from 380 to 4 hours in RMACC Summit and from 662 hours to 28 hours in Beocat); reduced turnaround time (from 403 to 6 hours in RMACC Summit and from 673 hours to 35 hours in Beocat); and acheived up to 100% utilization for both HPC resources.

All-gather Algorithms Resilient to Imbalanced Process Arrival Patterns

Two novel algorithms for the all-gather operation resilient to imbalanced process arrival patterns (PATs) are presented. The first one, Background Disseminated Ring (BDR), is based on the regular parallel ring algorithm often supplied in MPI implementations and exploits an auxiliary background thread for early data exchange from faster processes to accelerate the performed all-gather operation. The other algorithm, Background Sorted Linear synchronized tree with Broadcast (BSLB), is built upon the already existing PAP-aware gather algorithm, that is, Background Sorted Linear Synchronized tree (BSLS), followed by a regular broadcast distributing gathered data to all participating processes. The background of the imbalanced PAP subject is described, along with the PAP monitoring and evaluation topics. An experimental evaluation of the algorithms based on a proposed mini-benchmark is presented. The mini-benchmark was performed over 2,000 times in a typical HPC cluster architecture with homogeneous compute nodes. The obtained results are analyzed according to different PATs, data sizes, and process numbers, showing that the proposed optimization works well for various configurations, is scalable, and can significantly reduce the all-gather elapsed times, in our case, up to factor 1.9 or 47% in comparison with the best state-of-the-art solution.

Applying neural networks to predict HPC-I/O bandwidth over seismic data on lustre file system for ExSeisDat

HPC or super-computing clusters are designed for executing computationally intensive operations that typically involve large scale I/O operations. This most commonly involves using a standard MPI library implemented in C/C++. The MPI-I/O performance in HPC clusters tends to vary significantly over a range of configuration parameters that are generally not taken into account by the algorithm. It is commonly left to individual practitioners to optimise I/O on a case by case basis at code level. This can often lead to a range of unforeseen outcomes. The ExSeisDat utility is built on top of the native MPI-I/O library comprising of Parallel I/O and Workflow Libraries to process seismic data encapsulated in SEG-Y file format. The SEG-Y File data structure is complex in nature, due to the alternative arrangement of trace header and trace data. Its size scales to petabytes and the chances of I/O performance degradation are further increased by ExSeisDat. This research paper presents a novel study of the changing I/O performance in terms of bandwidth, with the use of parallel plots against various MPI-I/O, Lustre (Parallel) File System and SEG-Y File parameters. Another novel aspect of this research is the predictive modelling of MPI-I/O behaviour over SEG-Y File benchmarks using Artificial Neural Networks (ANNs). The accuracy ranges from 62.5% to 96.5% over the set of trained ANN models. The computed Mean Square Error (MSE), Mean Absolute Error (MAE) and Mean Absolute Percentage Error (MAPE) values further support the generalisation of the prediction models. This paper demonstrates that by using our ANNs prediction technique, the configurations can be tuned beforehand to avoid poor I/O performance.

Application of Container Technology in Numerical Ocean Model: a Kind of High-performance ROMS Containerized Architecture

Numerical ocean models are an important means for humans to study ocean activities today. For decades, with the continuous improvement of supercomputer performance, the application of numerical ocean models in high-performance computing has also developed rapidly. The birth and progress of high-performance cloud computing provide an opportunity for the development of portability and reproducibility of the ocean model. This paper uses container technology to construct the working environment of the Regional Oceanic Modeling System (ROMS), and rapidly completes the calculation jobs of the ocean model on the high-performance cloud computing platform. At the same time, we compare and analyze the operating performance of traditional HPC VM clusters. The new ocean model system we constructed based on high-performance cloud computing can provide better portability and convenience without wasting performance on traditional HPC clusters. It provides a new tool for scientists to quickly build ocean model clusters suitable for different hardware configurations, and to perform numerical ocean model calculations more conveniently.

Dataset for SC 21: Cross-Cluster User and Job Behavior on Production HPC Clusters

Dataset for SC 21: Cross-Cluster User and Job Behavior on Production HPC Clusters

빅데이터 분석 및 머신러닝 서비스를 위한 컨테이너 기반 HPC 클러스터 구축 및 성능 분석

기상, 화학, 고에너지 물리와 같은 기초 과학 분야의 대규모 워크로드를 실행하기 위해 대규모의 클러스터 시스템에서는 병렬 컴퓨팅에 최적화된 시스템 환경을 구축하여 서비스를 제공하고 있다. 그러나 최근 4차 산업 혁명으로 인하여 빅데이터 분석, 머신 러닝과 같은 새로운 형태의 사용자 요구 사항이 제시됨에 따라 기존 클러스터 시스템에서는 경직된 소프트웨어 구성, 유지 보수 비용 증가, 제한된 작업형태 지원 등 많은 어려움에 직면해 있다. 이러한 어려움을 해결하기 위해 본 논문에서는 컨테이너 기반 HPC 클러스터 구축 방안을 제시한다. 제시된 구축 방안에서는 컨테이너 기술을 활용하여 클러스터 시스템의 소프트웨어 스택을 간결화함으로써 자원 유연성을 향상시켜 사용자 맞춤형 서비스가 가능하다. 실험환경은 컨테이너 실행 데몬으로 Docker, 컨테이너 관리 툴킷으로는 kubernetes를 활용하여 구축했으며 HPL, HPCG, NPB benchmark 툴킷을 이용하여 다양한 워크로드에 대한 계산 성능 측정 결과를 제시한다.

SkePU 3: Portable High-Level Programming of Heterogeneous Systems and HPC Clusters

We present the third generation of the C++-based open-source skeleton programming framework SkePU. Its main new features include new skeletons, new data container types, support for returning multiple objects from skeleton instances and user functions, support for specifying alternative platform-specific user functions to exploit e.g. custom SIMD instructions, generalized scheduling variants for the multicore CPU backends, and a new cluster-backend targeting the custom MPI interface provided by the StarPU task-based runtime system. We have also revised the smart data containers’ memory consistency model for automatic data sharing between main and device memory. The new features are the result of a two-year co-design effort collecting feedback from HPC application partners in the EU H2020 project EXA2PRO, and target especially the HPC application domain and HPC platforms. We evaluate the performance effects of the new features on high-end multicore CPU and GPU systems and on HPC clusters.

Distributed CNN Inference on Resource-Constrained UAVs for Surveillance Systems: Design and Optimization

Unmanned Aerial Vehicles (UAVs) have attracted great interest in the last few years owing to their ability to cover large areas and access difficult and hazardous target zones, which is not the case of traditional systems relying on direct observations obtained from fixed cameras and sensors. Furthermore, thanks to the advancements in computer vision and machine learning, UAVs are being adopted for a broad range of solutions and applications. However, Deep Neural Networks (DNNs) are progressing toward deeper and complex models that prevent them from being executed on-board. In this paper, we propose a DNN distribution methodology within UAVs to enable data classification in resource-constrained devices and avoid extra delays introduced by the server-based solutions due to data communication over air-to-ground links. The proposed method is formulated as an optimization problem that aims to minimize the latency between data collection and decision-making while considering the mobility model and the resource constraints of the UAVs as part of the air-to-air communication. We also introduce the mobility prediction to adapt our system to the dynamics of UAVs and the network variation. The simulation conducted to evaluate the performance and benchmark the proposed methods, namely Optimal UAV-based Layer Distribution (OULD) and OULD with Mobility Prediction (OULD-MP), were run in an HPC cluster. The obtained results show that our optimization solution outperforms the existing and heuristic-based approaches.

Distributed in-memory data management for workflow executions.

Complex scientific experiments from various domains are typically modeled as workflows and executed on large-scale machines using a Parallel Workflow Management System (WMS). Since such executions usually last for hours or days, some WMSs provide user steering support, i.e., they allow users to run data analyses and, depending on the results, adapt the workflows at runtime. A challenge in the parallel execution control design is to manage workflow data for efficient executions while enabling user steering support. Data access for high scalability is typically transaction-oriented, while for data analysis, it is online analytical-oriented so that managing such hybrid workloads makes the challenge even harder. In this work, we present SchalaDB, an architecture with a set of design principles and techniques based on distributed in-memory data management for efficient workflow execution control and user steering. We propose a distributed data design for scalable workflow task scheduling and high availability driven by a parallel and distributed in-memory DBMS. To evaluate our proposal, we develop d-Chiron, a WMS designed according to SchalaDB’s principles. We carry out an extensive experimental evaluation on an HPC cluster with up to 960 computing cores. Among other analyses, we show that even when running data analyses for user steering, SchalaDB’s overhead is negligible for workloads composed of hundreds of concurrent tasks on shared data. Our results encourage workflow engine developers to follow a parallel and distributed data-oriented approach not only for scheduling and monitoring but also for user steering.

Parallel Fast Multipole Method accelerated FFT on HPC clusters

With increasing sizes of distributed systems, there comes an increased risk of communication bottlenecks. In the past decade there has been a growing interest in communication-avoiding algorithms. The distributed memory Fast Fourier Transform is an important algorithm which suffers from major communication bottlenecks. In this work, we take a look at an existing communication-avoiding algorithm FMM-FFT, an alternative to FFT which utilizes the Fast Multipole Method (FMM) to reduce communications to a single all-to-all communication. We present a detailed implementation of FMM-FFT relying on modern libraries and demonstrate it on two distinct distributed memory architectures notably a traditional Intel Xeon based HPC cluster and then a Beowulf cluster. We show that while the FMM-FFT is significantly slower than FFT on the traditional HPC cluster, on the Beowulf cluster it outperforms standard FFT, consistently getting speedups of 1.5x or more against FFTW. We then proceed to show how the communication to computation cost metric is important and useful in explaining the performance results of FMM-FFT against standard FFT. The source code pertaining to this work is being made publicly available under a permissive open source licence at Github.