HPC Clusters Research Articles

HPC Ecosystem and Competences in Bulgaria

In short, the current status of the HPC ecosystem in Bulgaria includes diverse modern hardware (including two supercomputers from the TOP500 list – Discoverer and HEMUS, as well as multiple HPC clusters), and extensive expertise in HPC and the adjacent fields. The competencies and expertise were acquired during a multitude of successful EU projects, as well as leadership and participation in high-profile national programmes and projects with industry/SMEs. Experience in negotiating and managing contracts with different sources of funding and for the use of both equipment and expertise has been gathered. In this paper, we discuss the new developments in the HPC Ecosystem, supported by the acquisition of the HEMUS supercomputer in 2023, as well as our design choices and optimization approaches.

Deploying Red Hat HPC Cluster in Al-Neelain University Labs

Deploying Red Hat HPC Cluster in Al-Neelain University Labs

Implementation and testing of a KNS topology in an InfiniBand cluster

The InfiniBand (IB) interconnection technology is widely used in the networks of modern supercomputers and data centers. Among other advantages, the IB-based network devices allow for building multiple network topologies, and the IB control software (subnet manager) supports several routing engines suitable for the most common topologies. However, the implementation of some novel topologies in IB-based networks may be difficult if suitable routing algorithms are not supported, or if the IB switch or NIC architectures are not directly applicable for that topology. This work describes the implementation of the network topology known as KNS in a real HPC cluster using an IB network. As far as we know, this is the first implementation of this topology in an IB-based system. In more detail, we have implemented the KNS routing algorithm in the OpenSM software distribution of the subnet manager, and we have adapted the available IB-based switches to the particular structure of this topology. We have evaluated the correctness of our implementation through experiments in the real cluster, using well-known benchmarks. The obtained results, which match the expected performance for the KNS topology, show that this topology can be implemented in IB-based clusters as an alternative to other interconnection patterns.

Pacybara: accurate long-read sequencing for barcoded mutagenized allelic libraries.

Long-read sequencing technologies, an attractive solution for many applications, often suffer from higher error rates. Alignment of multiple reads can improve base-calling accuracy, but some applications, e.g. sequencing mutagenized libraries where multiple distinct clones differ by one or few variants, require the use of barcodes or unique molecular identifiers. Unfortunately, sequencing errors can interfere with correct barcode identification, and a given barcode sequence may be linked to multiple independent clones within a given library. Here we focus on the target application of sequencing mutagenized libraries in the context of multiplexed assays of variant effects (MAVEs). MAVEs are increasingly used to create comprehensive genotype-phenotype maps that can aid clinical variant interpretation. Many MAVE methods use long-read sequencing of barcoded mutant libraries for accurate association of barcode with genotype. Existing long-read sequencing pipelines do not account for inaccurate sequencing or nonunique barcodes. Here, we describe Pacybara, which handles these issues by clustering long reads based on the similarities of (error-prone) barcodes while also detecting barcodes that have been associated with multiple genotypes. Pacybara also detects recombinant (chimeric) clones and reduces false positive indel calls. In three example applications, we show that Pacybara identifies and correctly resolves these issues. Pacybara, freely available at https://github.com/rothlab/pacybara, is implemented using R, Python, and bash for Linux. It runs on GNU/Linux HPC clusters via Slurm, PBS, or GridEngine schedulers. A single-machine simplex version is also available.

Cosmological constraints on neutrino mass within consistent cosmological models* *We acknowledge the use of HPC Cluster of ITP-CAS. XZ is supported by the National Natural Science Foundation of China (12005183). QGH is supported by the National Natural Science Foundation of China (12250010, 11975019, 11991052, 12047503), and the Key Research Program of Frontier Sciences, CAS, (ZDBS-LY-7009)

Recently, the emergence of cosmological tension has raised doubts about the consistency of the ΛCDM model. To constrain the neutrino mass within a consistent cosmological framework, we investigate three massive neutrinos with normal hierarchy (NH) and inverted hierarchy (IH) in both the axion-like early dark energy (Axi-EDE) and AdS-EDE models. We use joint datasets including the cosmic microwave background power spectrum from Planck 2018, Pantheon of type Ia supernova, baryon acoustic oscillation, and data from SH0ES. For the νAxi-EDE model, we obtain < 0.152 eV and < 0.178 eV, whereas for the νAdS-EDE model, we find < 0.135 eV and < 0.167 eV. Our results exhibit a preference for NH in both the νAxi-EDE and νAdS-EDE models.

Mid-fidelity NATO generic destroyer static and moving-ship airwake simulations using the Lattice-Boltzmann method compared with experimental data

During rotorcraft ship-deck landing operations, complex interactional aerodynamic phenomena occur between the rotor and ship airwakes which are not fully understood. To aid in ship–rotorcraft interactional investigations, efficient mid-fidelity ship airwake solutions are needed. Additionally, the effects of ship motion have not been explored much. In this study, the NATO Generic Destroyer, a shared, representative ship geometry created for collaborative studies, was investigated numerically using a GPU-accelerated Lattice-Boltzmann Method solver, and the ship airwake results compared with wind tunnel measurements using discrete velocity probes on the landing deck. The ship surface was modeled using a robust Grad immersed boundary approach, and ship motion was thereby included in the simulations. Ship airwake results showed good agreement with the experiment and excellent computational performance. Simulations took under four hours to run on a single GPU node. This is several orders of magnitude improvement over traditional Delayed Detached-Eddy Simulations typically used to solve this problem, which may take days or weeks on CPU-based HPC clusters.

Modular toolsets for integrating HPC clusters in experiment control systems

New particle/nuclear physics experiments require a massive amount of computing power that is only achieved by using high performance clusters directly connected to the data acquisition systems and integrated into the online systems of the experiments. However, integrating an HPC cluster into the online system of an experiment means: Managing and synchronizing thousands of processes that handle the huge throughput. In this work, modular components that can be used to build and integrate such a HPC cluster in the experiment control systems (ECS) will be introduced. The Online Device Control library (ODC) [1] in combination with the Dynamic Deployment System (DDS) [2, 3] and FairMQ [4] message queuing library offers a sustainable solution for integrating HPC cluster controls into an ECS. DDS as part of the ALFA framework [5] is a toolset that automates and significantly simplifies a dynamic deployment of user-defined processes and their dependencies on any resource management system (RMS) using a given process graph (topology). ODC, in this architecture, is the tool to control and communicate with a topology of FairMQ processes using DDS. ODC is designed to act as a broker between a high level experiment control system and a low level task management system e.g.: DDS. In this work the architecture of both DDS and ODC will be discussed, as well as the design decisions taken based on the experience gained of using these tools in production by the ALICE experiment at CERN to deploy and control thousands of processes (tasks) on the Event Processing Nodes cluster (EPN) during Run3 as a part of the ALICE O2 software ecosystem [6].

Simulator Demonstration of Large Scale Variational Quantum Algorithm on HPC Cluster

Simulator Demonstration of Large Scale Variational Quantum Algorithm on HPC Cluster

An overview of the Legio fault resilience framework for MPI applications

While the computational power of HPC clusters breached through the exascale milestone, it highlighted the need for some critical features like fault management. The current de facto standard for inter-process communication at scale, MPI, lacks proper fault management functionalities, precluding result production in the presence of faults. Many efforts have already addressed the problem, with the ULFM extension and Reinit proposal being the two works receiving the most attention. While powerful and effective, they require expertise from the user and significantly impact the application code. For this reason, we presented the Legio fault resilience framework, which can introduce graceful degradation properties in MPI applications with minimal changes. This work summarises the development and evolution of Legio, highlighting successful use cases and providing some tips for users who want to leverage it for their executions.

Salome_cfd: An HPC framework for thermalhydraulics and multiphysics powered by the open-source Computational Fluid Dynamics Toolbox code_saturne

EDF R&amp;D lab Chatou, Fluid Mechanics Energy and Environment department (MFEE), has been developping an HPC oriented Computational Fluid Dynamics toolbox since the late 1990s. This toolbox is based on the open-source Computational Fluid Dynamics (CFD) and multiphysics solver code_saturne. It has a proprietary extension, neptune_cfd, which is a Computational Multiphase Fluid Dynamics solver dealing with multi-phase flows. After a brief overview of code_saturne and neptune_cfd architecture and HPC capabilites, the present paper will focus on the effort made at EDF R&amp;D to harness HPC clusters for thermalhydraulics at an industrial scale. This is made available by codes built from the start with massively-parallel architectures in mind, thus leading to highly scalable solvers which are able to handle tens of billions of cells. User-friendly access and interoperability are pivotal for engineers to take advantage of these HPC capabilities. Among common applications requiring HPC resources, a focus is first made on frontier scale simulations, then on parametric studies and uncertainty quantifications experiments, where a twolevel parallelism is needed : a parallel Design of Experiment evaluation drives computations themselves (massively) parallel.

Electro-Catalytic Nitrogen Reduction by Metal Oxynitride Thin Films: Effect of Metal Oxophilicity, and Role of Lattice Nitrogen

The electrocatalytic reduction of molecular nitrogen to ammonia—the nitrogen reduction reaction (NRR)—is of broad interest as an environmentally- and energy-friendly alternative to the Haber-Bosch process for agricultural and emerging energy applications. Herein, we present our recent findings from collaborative electrochemistry/surface science/theoretical studies that counter several commonly held assumptions regarding transition metal oxynitrides and oxides as NRR catalysts. Specifically, we demonstrate that for the vanadium oxide, vanadium oxynitride, and cobalt oxynitride systems, (a) there is no Mars-van Krevelen mechanism and that the reduction of lattice nitrogen and N2 to NH3 occurs by parallel reaction mechanisms at O-ligated metal sites without incorporation of N into the oxide lattice, and (b) that NRR and the hydrogen evolution reaction (HER) do occur in concert under the conditions studied for Co oxynitride, but not for V oxynitride. Additionally, these results highlight the importance of both O-ligation of the V or Co center for the metal-binding of dinitrogen, and the importance of N in stabilizing the transition metal cation in an intermediate oxidation state for effective N≡N bond activation.1-3 Acknowledgment: This work was supported by the National Science Foundation through grants DMR-2112864 (to JAK, TRC, and FD) and partial support by CHE-1953547 (to TRC). Additional NSF support for the UNT CASCaM HPC cluster via Grant CHE-1531468 is also gratefully acknowledged. Ganesan, A. Osonkie, P. Chukwunenye, T. Cundari, F. D'Souza, J. Kelber. Electrochemical Reduction of N2 to ammonia by vanadium oxide thin films at neutral pH, J. Electrochem. Soc., 2021, 168, 026504.Osonkie, A. Ganesan, P. Chukwunenye, F. Anwar, K. Balogun, M. Gharee, I. Rashed, T. R. Cundari, F. D’Souza, and J. A. Kelber, Electro-catalytic reduction of nitrogen to ammonia: The roles of lattice O and N in reduction at vanadium oxynitride surfaces, ACS Adv. Mater & Interface, 2022, 14, 531-542.Chukwunenye, A. Ganesan, M. Gharaee, K. Balogun, F. Anwar, Q. Adesope, T. R. Cundari, F. D’Souza, and J. A. Kelber, ‘Electrocatalytic Selectivity for Nitrogen Reduction vs. Hydrogen Evolution: A Comparison of Vanadium and Cobalt Oxynitrides at Different pH Values.’ J. Mater. Chem. A, 2022, 10, 21401-21415.

Which Interfaces Matter Most? Variability in Grain Boundary Defect Chemistry and Conductivity in a Concentrated Solid Electrolyte

Inorganic solid electrolytes with fast ionic conductivity are critical components of fuel/electrolysis cells, all-solid-state batteries, membrane and separation reactors, sensors and other electrochemical devices. For example, solid oxygen ion (O2-) conductors are used in solid oxide fuel/electrolysis cells (SOFCs/SOECs) for high-efficiency conversion between chemical and electrical energy and chemical production from CO2. One major challenge in engineering such materials is the diminished grain boundary (GB) conductivity in polycrystalline oxide ceramics. This phenomenon, which was initially accredited to highly resistive silica-based glassy impurity phases, is attributed to the formation of intrinsic space charge layers (SCLs) directly adjacent to GB cores. The positive potential at the GB core (due to large concentrations of oxygen vacancies) results in a depletion of oxygen vacancies in the space charge layer as well as the segregation of charge-compensating acceptor solutes from grain interiors which ultimately reduces GB conductivity. Several strategies have been explored to improve GB conductivity by understanding and mitigating the SCL effects. While the standard Poisson–Boltzmann approach models the near-grain boundary defect behavior in dilute solid solutions and recent models by Mebane et al [1] and Virkant et al [2] incorporate the effects of defect-defect interactions (in concentrated solid solutions) and chemomechanical stress, respectively, there is yet to be a model developed that can predict individual GB conductivities in concentrated solid solutions based on direct experimental measurements of near-GB defect chemistry.Here, we present an experimental-computational framework that predicts which GBs in a polycrystalline electrolyte likely facilitate ionic conductivity. We developed a thermodynamic phase-field modeling framework and applied it to a model oxygen electrolyte Gd0.25Ce0.75O2-δ, wherein the GB-to-GB variability in defect concentrations (Gd3+ solutes, electrons localized at Ce3+, and oxygen vacancies) was measured microscopically using scanning transmission electron microscopy electron energy-loss spectroscopy (STEM EELS). These data were used to predict the GB-to-GB variability in cross-GB ionic conduction using a unique model, which prioritizes reproducing microscopically observed GB defect concentrations and considers defect-defect interactions in highly concentrated solid solutions, making the framework applicable to other technologically relevant solid electrolytes. Across the GBs studied, we revealed a non-monotonic relationship between depletion and GB conductivity, with the highest conductivity predicted for intermediate depletion amounts. This work lays the foundation for future experimental-computational research and advanced data-driven design of solid electrolytes needed for functional and energy storage/conversion devices.[1] D. S. Mebane, R. A. D. Souza, Energy & Environmental Science, 2015.[2] K. S. N. Vikrant, R. Edwin García, Energy & Environmental Science, 2018.[3] H. Vahidi, W.J. Bowman (Submitted). Acknowledgments HV and WJB acknowledge primary support from the UCI new faculty startup funding. This research was partially supported by the National Science Foundation Materials Research Science and Engineering Center program through the UC Irvine Center for Complex and Active Materials (DMR-2011967). We acknowledge the use of facilities and instrumentation at the UC Irvine Materials Research Institute (IMRI) supported in part by the same NSF MRSEC (DMR-2011967). AM, DM, and AC acknowledge funding from the NSF [WB1] [hv2] and the WVU MAE research department for the use of the Thorny Flat HPC Cluster for the numerical modeling work and computational resources.

A Fast, Reproducible, High-throughput Variant Calling Workflow for Population Genomics.

The increasing availability of genomic resequencing data sets and high-quality reference genomes across the tree of life present exciting opportunities for comparative population genomic studies. However, substantial challenges prevent the simple reuse of data across different studies and species, arising from variability in variant calling pipelines, data quality, and the need for computationally intensive reanalysis. Here, we present snpArcher, a flexible and highly efficient workflow designed for the analysis of genomic resequencing data in nonmodel organisms. snpArcher provides a standardized variant calling pipeline and includes modules for variant quality control, data visualization, variant filtering, and other downstream analyses. Implemented in Snakemake, snpArcher is user-friendly, reproducible, and designed to be compatible with high-performance computing clusters and cloud environments. To demonstrate the flexibility of this pipeline, we applied snpArcher to 26 public resequencing data sets from nonmammalian vertebrates. These variant data sets are hosted publicly to enable future comparative population genomic analyses. With its extensibility and the availability of public data sets, snpArcher will contribute to a broader understanding of genetic variation across species by facilitating the rapid use and reuse of large genomic data sets.

Two different parallel approaches for a hybrid fractional order Coronavirus model

In this paper, two different parallel approaches for a hybrid fractional order Coronavirus (2019-nCov) mathematical model are presented. Both parallel approaches are implemented using Julia high level language. Parallel algorithm implementations are developed for the HPC cluster using Message Passing Interface (MPI) technology and general-purpose computing on GPUs (GPGPU) using Compute Unified Device Architecture (CUDA) based on hardware environments. The algorithm implementation are used to solve the real-world problem of the hybrid fractional order Coronavirus (2019-nCov) mathematical model and to study the parallel efficiency. The introduced hybrid fractional order derivative is defined as a linear combination of the integral of Riemann-Liouville and the fractional order Caputo derivative. A parallel algorithm is designed based on the predictor-corrector method with the discretization of the Caputo proportional constant fractional hybrid operator for solving the model problem numerically. Simulation results show that, both the new parallel approaches achieve significant efficiency.

Top-Down Models across CPU Architectures: Applicability and Comparison in a High-Performance Computing Environment

Top-Down models are defined by hardware architects to provide information on the utilization of different hardware components. The target is to isolate the users from the complexity of the hardware architecture while giving them insight into how efficiently the code uses the resources. In this paper, we explore the applicability of four Top-Down models defined for different hardware architectures powering state-of-the-art HPC clusters (Intel Skylake, Fujitsu A64FX, IBM Power9, and Huawei Kunpeng 920) and propose a model for AMD Zen 2. We study a parallel CFD code used for scientific production to compare these five Top-Down models. We evaluate the level of insight achieved, the clarity of the information, the ease of use, and the conclusions each allows us to reach. Our study indicates that the Top-Down model makes it very difficult for a performance analyst to spot inefficiencies in complex scientific codes without delving deep into micro-architecture details.

OpenFAM: Programming disaggregated memory

AbstractHigh performance computing (HPC) clusters are increasingly handling workloads where working data sets cannot be easily partitioned or are too large to fit into local node memory. In order to enable HPC workloads to access memory external to the node, HPE has defined a programming API (OpenFAM) for developing applications that use large‐scale disaggregated memory. In this paper we describe an open‐source reference implementation of OpenFAM that can be used on scale‐up machines, traditional HPC clusters, as well as emerging disaggregated memory architectures. We demonstrate the efficiency of the implementation using micro‐benchmarks on InfiniBand and Slingshot‐based clusters.

MD-Ligand-Receptor: A High-Performance Computing Tool for Characterizing Ligand-Receptor Binding Interactions in Molecular Dynamics Trajectories.

Molecular dynamics simulation is a widely employed computational technique for studying the dynamic behavior of molecular systems over time. By simulating macromolecular biological systems consisting of a drug, a receptor and a solvated environment with thousands of water molecules, MD allows for realistic ligand-receptor binding interactions (lrbi) to be studied. In this study, we present MD-ligand-receptor (MDLR), a state-of-the-art software designed to explore the intricate interactions between ligands and receptors over time using molecular dynamics trajectories. Unlike traditional static analysis tools, MDLR goes beyond simply taking a snapshot of ligand-receptor binding interactions (lrbi), uncovering long-lasting molecular interactions and predicting the time-dependent inhibitory activity of specific drugs. With MDLR, researchers can gain insights into the dynamic behavior of complex ligand-receptor systems. Our pipeline is optimized for high-performance computing, capable of efficiently processing vast molecular dynamics trajectories on multicore Linux servers or even multinode HPC clusters. In the latter case, MDLR allows the user to analyze large trajectories in a very short time. To facilitate the exploration and visualization of lrbi, we provide an intuitive Python notebook (Jupyter), which allows users to examine and interpret the results through various graphical representations.

Making applications faster by asynchronous execution: Slowing down processes or relaxing MPI collectives

Comprehending the performance bottlenecks at the core of the intricate hardware–software interactions exhibited by highly parallel programs on HPC clusters is crucial. This paper sheds light on the issue of automatically asynchronous MPI communication in memory-bound parallel programs on multicore clusters and how it can be facilitated. For instance, slowing down MPI processes by deliberate injection of delays can improve performance if certain conditions are met. This leads to the counter-intuitive conclusion that noise, independent of its source, is not always detrimental but can be leveraged for performance improvements. We employ phase-space graphs as a new tool to visualize parallel program dynamics. They are useful in spotting certain patterns in parallel execution that will easily go unnoticed with traditional tracing tools. We investigate five different microbenchmarks and applications on different supercomputer platforms: an MPI-augmented STREAM Triad, two implementations of Lattice-Boltzmann fluid solvers (D3Q19 and SPEChpc D2Q37), the LULESH and HPCG proxy applications.

POETS: An Event-driven Approach to Dissipative Particle Dynamics

HPC clusters have become ever more expensive, both in terms of capital cost and energy consumption; some estimates suggest that competitive installations at the end of the next decade will require their own power station. One way around this looming problem is to design bespoke computing engines, but while the performance benefits are good, the design costs are huge and cannot easily be amortized. Partially Ordered Event Triggered System (POETS)—the focus of this article—seeks to exploit a middle way: The architecture is tuned to a specific algorithmic pattern but, within that constraint, is fully programmable. POETS software is quasi-imperative: The user defines a set of sequential event handlers, defines the topology of a (typically large) concurrent ensemble of these, and lets them interact. The “solution” may be exfiltrated from the emergent behaviour of the ensemble. In this article, we describe (briefly) the architecture, and an example computational chemistry application, dissipative particle dynamics (DPD). The DPD algorithm is traditionally implemented using parallel computational techniques, but we re-cast it as a concurrent compute problem that is then ideally suited to POETS. Our prototype system is realised on a cluster of 48 FPGAs providing 50K concurrent hardware threads, and we report performance speedups of over two orders of magnitude better than a single thread baseline comparator and scaling behaviour that is almost constant. The results are validated against a “conventional” implementation.

Job runtime prediction of HPC cluster based on PC-Transformer

Job runtime prediction of HPC cluster based on PC-Transformer