High Performance Computing Environments Research Articles

Graph and convolutional neural network coupling with a high-performance large-eddy simulation solver

Computational Fluid Dynamics (CFD) traditionally relies on long-standing numerical simulation strategies for the Navier–Stokes equations. Recently, interest in data-driven hybrid CFD solvers has spiked, leveraging pre-computed datasets to enhance various weak links inside existing solvers, such as closure models, under-resolved physics, or even to guide numerical resolution strategies. Running these hybrid solvers, notably in High Performance Computing (HPC) environments, presents specific challenges. In particular, context-aware deep learning (e.g. Convolutional (CNN) and Graph (GNN) Neural Networks) is promising for this task, but requires passing data representations between the physics solver and the neural network. In relevant industrial configurations, CFD meshes can be Cartesian but highly irregular, or unstructured, both of which do not match the pixel/voxel structure needed to run CNNs. In addition, discrepancies in programming language and libraries are common between CFD and machine learning applications. This work explores the many challenges of running a parallel hybrid solver in an HPC context, through the coupling of the AVBP CFD solver with neural networks in turbulent combustion and wall friction modeling applications. The knowledge gained is showcased in this article, as well as assembled in an actionable open-source library.

A pragma based C++ framework for hybrid quantum/classical computation

Quantum computers promise exponential speed ups over classical computers for various tasks. This emerging technology is expected to have its first huge impact in High Performance Computing (HPC), as it can solve problems beyond the reach of HPC. To that end, HPC will require quantum accelerators, which will enable applications to run on both classical and quantum devices, via hybrid quantum-classical nodes. Hybrid quantum-HPC applications should be scalable, executable on Quantum Error Corrected (QEC) devices, and could use quantum-classical primitives. However, the lack of scalability, poor performances, and inability to insert classical schemes within quantum applications has prevented current quantum frameworks from being adopted by the HPC community.This paper specifies the requirements of a hybrid quantum-classical framework compatible with HPC environments, and introduces a novel hardware-agnostic framework called Q-Pragma. This framework extends the classical programming language C++ heavily used in HPC via the addition of pragma directives to manage quantum computations.

ViralFlow v1.0-a computational workflow for streamlining viral genomic surveillance.

ViralFlow v1.0 is a computational workflow developed for viral genomic surveillance. Several key changes turned ViralFlow into a general-purpose reference-based genome assembler for all viruses with an available reference genome. New virus-agnostic modules were implemented to further study nucleotide and amino acid mutations. ViralFlow v1.0 runs on a broad range of computational infrastructures, from laptop computers to high-performance computing (HPC) environments, and generates standard and well-formatted outputs suited for both public health reporting and scientific problem-solving. ViralFlow v1.0 is available at: https://viralflow.github.io/index-en.html.

MaPPeRTrac: A Massively Parallel, Portable, and Reproducible Tractography Pipeline.

Large-scale diffusion MRI tractography remains a significant challenge. Users must orchestrate a complex sequence of instructions that requires many software packages with complex dependencies and high computational costs. We developed MaPPeRTrac, an edge-centric tractography pipeline that simplifies and accelerates this process in a wide range of high-performance computing (HPC) environments. It fully automates either probabilistic or deterministic tractography, starting from a subject's magnetic resonance imaging (MRI) data, including structural and diffusion MRI images, to the edge density image (EDI) of their structural connectomes. Dependencies are containerized with Singularity (now called Apptainer) and decoupled from code to enable rapid prototyping and modification. Data derivatives are organized with the Brain Imaging Data Structure (BIDS) to ensure that they are findable, accessible, interoperable, and reusable following FAIR principles. The pipeline takes full advantage of HPC resources using the Parsl parallel programming framework, resulting in the creation of connectome datasets of unprecedented size. MaPPeRTrac is publicly available and tested on commercial and scientific hardware, so it can accelerate brain connectome research for a broader user community. MaPPeRTrac is available at: https://github.com/LLNL/mappertrac .

ESSENCE-Dock: A Consensus-Based Approach to Enhance Virtual Screening Enrichment in Drug Discovery.

Drug development is a complex, costly, and time-consuming endeavor. While high-throughput screening (HTS) plays a critical role in the discovery stage, it is one of many factors contributing to these challenges. In certain contexts, virtual screening can complement the HTS, potentially offering a more streamlined approach in the initial stages of drug discovery. Molecular docking is an example of a popular virtual screening technique that is often used for this purpose; however, its effectiveness can vary greatly. This has led to the use of consensus docking approaches that combine results from different docking methods to improve the identification of active compounds and reduce the occurrence of false positives. However, many of these methods do not fully leverage the latest advancements in molecular docking. In response, we present ESSENCE-Dock (Effective Structural Screening ENrichment ConsEnsus Dock), a new consensus docking workflow aimed at decreasing false positives and increasing the discovery of active compounds. By utilizing a combination of novel docking algorithms, we improve the selection process for potential active compounds. ESSENCE-Dock has been made to be user-friendly, requiring only a few simple commands to perform a complete screening while also being designed for use in high-performance computing (HPC) environments.

Parallel Processing of Image Databases for Accelerated Morphological Analysis

Nuclear forensics relies on different signatures to identify the source of nuclear material. Such signatures include crystalline structure, chemical composition, and particle morphology. One way to quantify morphology in electron microscope imagery is through image segmentation, where pixels are assigned to several partitions (or groups) that correspond to particles, grains, and other objects of interest within the image. Once pixels are assigned to segments, it is relatively straightforward to quantify other quantities of interest, such as grain size, circularity, etc. However, the range and diversity of microscope images make it difficult to obtain an accurate segmentation automatically. The accuracy of segmentation can be improved through supervised learning, but this requires many images to be manually segmented. Another way to improve the accuracy is to use interactive segmentation. Interactive segmentation requires a human to provide image-specific user input to improve performance. However, the amount of user input (effort) is generally far less than is required for supervised learning. In this paper, we investigate several parallelization strategies to automatically explore the user input parameter space of interactive segmentation algorithms across a large number of images. Specifically, we investigate four different parallelization mechanisms in a high-performance computing (HPC) environment and use the Amdahl fraction to evaluate efficiency on multiple processor cores across multiple nodes. Ultimately, the parallelization strategy that was most efficient utilized the message passing interface integrated with the segmentation and quantification code. This strategy had an Amdahl fraction of 0.985 and a performance of about 0.251 s/image. These results indicate that the parameter space of interactive segmentation algorithms can be efficiently explored using HPC. This opens the door to future work where user input is reduced and where interactive image segmentation algorithms are automatically applied to large image sets.

Cloud-Based Infrastructure and DevOps for Energy Fault Detection in Smart Buildings

We have designed a real-world smart building energy fault detection (SBFD) system on a cloud-based Databricks workspace, a high-performance computing (HPC) environment for big-data-intensive applications powered by Apache Spark. By avoiding a Smart Building Diagnostics as a Service approach and keeping a tightly centralized design, the rapid development and deployment of the cloud-based SBFD system was achieved within one calendar year. Thanks to Databricks’ built-in scheduling interface, a continuous pipeline of real-time ingestion, integration, cleaning, and analytics workflows capable of energy consumption prediction and anomaly detection was implemented and deployed in the cloud. The system currently provides fault detection in the form of predictions and anomaly detection for 96 buildings on an active military installation. The system’s various jobs all converge within 14 min on average. It facilitates the seamless interaction between our workspace and a cloud data lake storage provided for secure and automated initial ingestion of raw data provided by a third party via the Secure File Transfer Protocol (SFTP) and BLOB (Binary Large Objects) file system secure protocol drivers. With a powerful Python binding to the Apache Spark distributed computing framework, PySpark, these actions were coded into collaborative notebooks and chained into the aforementioned pipeline. The pipeline was successfully managed and configured throughout the lifetime of the project and is continuing to meet our needs in deployment. In this paper, we outline the general architecture and how it differs from previous smart building diagnostics initiatives, present details surrounding the underlying technology stack of our data pipeline, and enumerate some of the necessary configuration steps required to maintain and develop this big data analytics application in the cloud.

Model Performance Prediction for Hyperparameter Optimization of Deep Learning Models Using High Performance Computing and Quantum Annealing

Hyperparameter Optimization (HPO) of Deep Learning (DL)-based models tends to be a compute resource intensive process as it usually requires to train the target model with many different hyperparameter configurations. We show that integrating model performance prediction with early stopping methods holds great potential to speed up the HPO process of deep learning models. Moreover, we propose a novel algorithm called Swift-Hyperband that can use either classical or quantum Support Vector Regression (SVR) for performance prediction and benefit from distributed High Performance Computing (HPC) environments. This algorithm is tested not only for the Machine-Learned Particle Flow (MLPF), model used in High-Energy Physics (HEP), but also for a wider range of target models from domains such as computer vision and natural language processing. Swift-Hyperband is shown to find comparable (or better) hyperparameters as well as using less computational resources in all test cases.

Achieving better Resource Utilization by Implementing a High-Performance Intelligent Framework in a Distributed Environment

Multi-distributed high-performance computers from many companies are aggregated into a single computing platform to provide handlers with uniform contact besides convention outlines. Job arrangement strategies in High-Performance Computing (HPC) environments are lacking in flexibility, so an enhanced computational intelligence automated system in the task ready queue, refinement of the principal planner aimed at every job, and increased arrangement of the job setting up plan are proposed in this paper, which introduces an improved task scheduling model. The swarm intelligence method is used in core task scheduling to reduce the average scheduling time for completing tasks by assigning jobs to each node in the most efficient manner possible. The suggested scheduling technique outperforms the standard work scheduling approach in simulations. Task waiting times can be reduced, system throughput increased, task response times improved, and system resources better utilized by using a job setting up method created on group Acumen systems

Investigating performance metrics for container-based HPC environments using x86 and OpenPOWER systems

Container-based High-Performance Computing (HPC) is changing the way computation is performed and reproduced without sacrificing the raw performance compared to hypervisor-assisted virtualization technologies. It primarily supports continuously evolving data-intensive applications such as computational fluid dynamics, seismic tomography, molecular biology, and Proteomics. OpenPOWER systems, unlike the x86 systems, use the POWER-compliant processor to exploit instruction-level and thread-level parallelism heavily. In our previous work, we designed and developed a Containerized HPC environment (cHPCe) from the scratch using Linux namespaces on OpenPOWER systems. This paper aims to provide an in-depth performance analysis of the Containerized HPC environment using x86 systems and Containerized HPC environment using the OpenPOWER system, on systems’ subcomponents, processor, memory, interconnect, and IO. This sub-component analysis provides an insight on several aspects of the system performance. To the best of our knowledge, no research has been reported yet for such a comparative analysis that designs cHPCe for both x86 and OpenPOWER systems. The performance of the developed cHPCe is compared with BareMetals, and VMs using the benchmarks HPCC, and IOZone. Our experimental results achieve 0.13% less compute performance penalty at its peak performance on cHPCe compared to the BareMetal-based solution for x86 systems. In contrast, a VM-based solution introduces an overhead of 20% and 4.83% in x86 and OpenPOWER cases, respectively. Moreover, the x86 and OpenPOWER systems observe inconsistent behavior for memory performance with a worst-case penalty of 9.68% and 6.64% compared to achieved peak performance, respectively. However, similar behavior is reported for cHPCe with an overhead of less than 3% and 2% in the worst case for the latency and bandwidth, respectively, compared to the BareMetal for network and disk performance. Our experimental results reveal that the containerized OpenPOWER environment represents a viable alternative to the counterpart x86 environment for the HPC solution.

Towards bespoke optimizations of energy efficiency in HPC environments

AbstractWe present bespoke energy efficiency optimizations in high performance computing (HPC) environments using holistic approach to data collection, analysis and proactive management of resources and workloads. Our solution has three major components: (i) platform for collecting, storing and processing data from multiple sources across hardware and software stacks, (ii) collections of regression machine learning (ML) algorithms for making workloads classifications and energy usage predictions, (iii) agent‐based decision‐making framework for delivering control decisions to middle‐ware and infrastructure thus supporting real time or near real energy efficiency optimizations. We will provide some concrete examples of using our proposed approach in HPC environment.

Omics Analysis and Quality Control Pipelines in a High-Performance Computing Environment.

Data quality is often an overlooked feature in the analysis of omics data. This is particularly relevant in studies of chemical and pathogen exposures that can modify an individual's epigenome and transcriptome with persistence over time. Portable, quality control (QC) pipelines for multiple different omics datasets are therefore needed. To meet these goals, portable quality assurance (QA) metrics, metric acceptability criterion, and pipelines to compute these metrics were developed and consolidated into one framework for 12 different omics assays. Performance of these QA metrics and pipelines were evaluated on human data generated by the Defense Advanced Research Projects Agency (DARPA) Epigenetic CHaracterization and Observation (ECHO) program. Twelve analytical pipelines were developed leveraging standard tools when possible. These QC pipelines were containerized using Singularity to ensure portability and scalability. Datasets for these 12 omics assays were analyzed and results were summarized. The quality thresholds and metrics used were described. We found that these pipelines enabled early identification of lower quality datasets, datasets with insufficient reads for additional sequencing, and experimental protocols needing refinements. These omics data analysis and QC pipelines are available as open-source resources as reported and discussed in this article for the omics and life sciences communities.

Opportunities for enhancing MLCommons efforts while leveraging insights from educational MLCommons earthquake benchmarks efforts

MLCommons is an effort to develop and improve the artificial intelligence (AI) ecosystem through benchmarks, public data sets, and research. It consists of members from start-ups, leading companies, academics, and non-profits from around the world. The goal is to make machine learning better for everyone. In order to increase participation by others, educational institutions provide valuable opportunities for engagement. In this article, we identify numerous insights obtained from different viewpoints as part of efforts to utilize high-performance computing (HPC) big data systems in existing education while developing and conducting science benchmarks for earthquake prediction. As this activity was conducted across multiple educational efforts, we project if and how it is possible to make such efforts available on a wider scale. This includes the integration of sophisticated benchmarks into courses and research activities at universities, exposing the students and researchers to topics that are otherwise typically not sufficiently covered in current course curricula as we witnessed from our practical experience across multiple organizations. As such, we have outlined the many lessons we learned throughout these efforts, culminating in the need for benchmark carpentry for scientists using advanced computational resources. The article also presents the analysis of an earthquake prediction code benchmark while focusing on the accuracy of the results and not only on the runtime; notedly, this benchmark was created as a result of our lessons learned. Energy traces were produced throughout these benchmarks, which are vital to analyzing the power expenditure within HPC environments. Additionally, one of the insights is that in the short time of the project with limited student availability, the activity was only possible by utilizing a benchmark runtime pipeline while developing and using software to generate jobs from the permutation of hyperparameters automatically. It integrates a templated job management framework for executing tasks and experiments based on hyperparameters while leveraging hybrid compute resources available at different institutions. The software is part of a collection called cloudmesh with its newly developed components, cloudmesh-ee (experiment executor) and cloudmesh-cc (compute coordinator).

Top-Down Models across CPU Architectures: Applicability and Comparison in a High-Performance Computing Environment

Top-Down models are defined by hardware architects to provide information on the utilization of different hardware components. The target is to isolate the users from the complexity of the hardware architecture while giving them insight into how efficiently the code uses the resources. In this paper, we explore the applicability of four Top-Down models defined for different hardware architectures powering state-of-the-art HPC clusters (Intel Skylake, Fujitsu A64FX, IBM Power9, and Huawei Kunpeng 920) and propose a model for AMD Zen 2. We study a parallel CFD code used for scientific production to compare these five Top-Down models. We evaluate the level of insight achieved, the clarity of the information, the ease of use, and the conclusions each allows us to reach. Our study indicates that the Top-Down model makes it very difficult for a performance analyst to spot inefficiencies in complex scientific codes without delving deep into micro-architecture details.

Python-based social science applications’ profiling and optimization on HPC systems using task and data parallelism

This research addresses the pressing need to optimize Python-based social science applications for high-performance computing (HPC)systems, emphasizing the combined use of task and data parallelism techniques. The paper delves into a substantial body of research,recognizing Python’s interpreted nature as a challenge for efficient social science data processing. The paper introduces a Pythonprogram that exemplifies the proposed methodology. This program uses task parallelism with multi-processing and data parallelismwith dask to optimize data processing workflows. It showcases how researchers can effectively manage large datasets and intricatecomputations on HPC systems. The research offers a comprehensive framework for optimizing Python-based social science applicationson HPC systems. It addresses the challenges of Python’s performance limitations, data-intensive processing, and memory efficiency.Incorporating insights from a rich literature survey, it equips researchers with valuable tools and strategies for enhancing the efficiencyof their social science applications in HPC environments.

Rapid and Automated Ab Initio Metabolite Collisional Cross Section Prediction from SMILES Input.

We implemented an ab initio CCS prediction workflow which incrementally refines generated structures using molecular mechanics, a deep learning potential, conformational clustering, and quantum mechanics (QM). Automating intermediate steps for a high performance computing (HPC) environment allows users to input the SMILES structure of small organic molecules and obtain a Boltzmann averaged collisional cross section (CCS) value as output. The CCS of a molecular species is a metric measured by ion mobility spectrometry (IMS) which can improve annotation of untargeted metabolomics experiments. We report only a minor drop in accuracy when we expedite the CCS calculation by replacing the QM geometry refinement step with a single-point energy calculation. Even though the workflow involves stochastic steps (i.e., conformation generation and clustering), the final CCS value was highly reproducible for multiple iterations on L-carnosine. Finally, we illustrate that the gas phase ensembles modeled for the workflow are intermediate files which can be used for the prediction of other properties such as aqueous phase nuclear magnetic resonance chemical shift prediction. The software is available at the following link: https://github.com/DasSusanta/snakemake_ccs.

An Optimal Network-Aware Scheduling Technique for Distributed Deep Learning in Distributed HPC Platforms

Deep learning is a growing technique used to solve complex artificial intelligence (AI) problems. Large-scale deep learning has become a significant issue as a result of the expansion of datasets and the complexity of deep learning models. For training large-scale models, the cloud can be used as a distributed HPC (high-performance computing) tool with benefits in cost and flexibility. However, one of the major performance barriers in distributed deep learning in a distributed HPC environment is the network. The performance is often limited by heavy traffic like many stochastic gradient descent transfers for distributed communication. There are many network studies in distributed deep learning to solve these problems, but most research only focuses on improving communication performance and applying new methods or algorithms like overlapping parameter synchronization to minimize communication delay rather than considering the actual network. In this paper, we are focusing on the actual network, especially in a distributed HPC environment. In such an environment, if cluster nodes are assigned to different zones/regions which means a set of an appropriate number of distributed HPC nodes when performing distributed deep learning tasks, performance degradation due to network delay may occur. The proposed network optimization algorithm ensures that distributed work is placed in the same zone as much as possible to reduce network delay. Furthermore, scoring using network monitoring tools like loss, delay, and throughput is applied to select the optimal node within the zone. Our proposal has been validated on the Kubernetes platform, an open source orchestrator for the automatic management and deployment of micro-services. The performance of distributed deep learning is improved through the proposed scheduler.

Dagster: Parallel Structured Search

We demonstrate Dagster, a system that implements a new approach to scheduling interdependent (Boolean) SAT search activities in high-performance computing (HPC) environments. Our system takes as input a set of disjunctive clauses (i.e., DIMACS CNF) and a labelled directed acyclic graph (DAG) structure describing how the clauses are decomposed into a set of interrelated problems. Component problems are solved using standard systematic backtracking search, which may optionally be coupled to (stochastic dynamic) local search and/or clause-strengthening processes. We demonstrate Dagster using a new Graph Maximal Determinant combinatorial case study. This demonstration paper presents a new case study, and is adjunct to the longer accepted manuscript at the Pacific Rim International Conference on Artificial Intelligence (2022).

Accelerated nanopore basecalling with SLOW5 data format.

Nanopore sequencing is emerging as a key pillar in the genomic technology landscape but computational constraints limiting its scalability remain to be overcome. The translation of raw current signal data into DNA or RNA sequence reads, known as 'basecalling', is a major friction in any nanopore sequencing workflow. Here, we exploit the advantages of the recently developed signal data format 'SLOW5' to streamline and accelerate nanopore basecalling on high-performance computing (HPC) and cloud environments. SLOW5 permits highly efficient sequential data access, eliminating a potential analysis bottleneck. To take advantage of this, we introduce Buttery-eel, an open-source wrapper for Oxford Nanopore's Guppy basecaller that enables SLOW5 data access, resulting in performance improvements that are essential for scalable, affordable basecalling. Buttery-eel is available at https://github.com/Psy-Fer/buttery-eel.

Assessing Saiph, a task-based DSL for high-performance computational fluid dynamics

Scientific applications face the challenge of efficiently exploiting increasingly complex parallel and distributed systems. Developing hand-tuned codes is a time-consuming, tedious and hardly reusable task. Reaching high performance appears detrimental to productivity and portability and unreasonable to expect from scientists. Domain-Specific Languages (DSLs) are collaborative environments aiming to overcome such difficulties by decoupling the problem description from the algorithmic implementation. However, developing a competitive tool in High-Performance Computing (HPC) is challenging: DSLs for HPC environments have two additional critical requirements, performance and scalability. Moreover, documented and successful cases are few, making it difficult to popularise DSLs as problem-solving environments for scientific HPC code development. In this context, Saiph is a task-based DSL easing the simulation of physical phenomena from Computational Fluid Dynamics (CFD), developed to meet HPC productivity and performance requirements. This work reports the tuning and evaluation of Saiph using the Taylor–Green Vortex (TGV) problem as a case study. We assess Saiph’s productivity, numerical methods, and high-performance strategies to illustrate its use and demonstrate its competitiveness, viability and benefits for CFD software developments in HPC environments. Hence, we contribute to the popularisation of HPC DSLs as suitable problem-solving environments able to unify modern computational and scientific knowledge.