Parallel Performance Research Articles

Scale-resolving simulation of a low-pressure turbine on hybrid supercomputers

The increased computing power of modern hybrid supercomputers is expanding the applicability of scale-resolving simulations in scientific and industrial applications. This paper is devoted to such a supercomputer simulation technology for turbomachinery. A heterogeneous parallel algorithm is outlined and its parallel performance is demonstrated on various hybrid supercomputers by executing a single simulation on dozens of central and graphics processing units. The scale-resolving numerical simulation of the flow in a linear cascade of T106C high-lift low-pressure turbine blades is considered. The presence of a laminar–turbulent transition on the suction side of the blade, caused by the separation of the boundary layer, leads to a significant increase in the loss of kinetic energy. The effect of increasing losses with a decrease in the Reynolds number is captured. A strong dependence of the numerical results on the incoming turbulent flow conditions is observed. The effect of imposing synthetic turbulence at the inflow is studied, as well as the required mesh resolution to capture the laminar–turbulent transition well.

Billion-design-variable-scale topology optimization of vehicle frame structure in multiple-load case

In topology optimization, sufficient resolution and a constraint volume of less than 1% are required to obtain a practical vehicle body structure without solid circular-section frames. To meet the requirement for sufficient resolution, the authors are developing voxel topology optimization software, including a finite element solver that utilizes the building cube method framework available in massively parallel environments. The authors have performed a topology optimization of billions of elements intended for a vehicle frame using 35,000–66,000 processors and measured its parallel performance. In addition, four different methods to treat multiple-load cases required for vehicle performance into single objective functions are examined. As a result, normalizing compliance with the appropriate target energy obtained by the original body-in-white frame balances the optimization performance across cases. In the single-load case, thick solid beams are generated through optimization. In contrast, such solid frames are suppressed in multiple-load cases, resulting in a structure similar to a practical body-in-white frame.

Assessment of Numerical Accuracy and Parallel Performance of OpenFOAM and its Reacting Flow Extension EBIdnsFoam

OpenFOAM is one of the most widely used open-source computational fluid dynamics tools and often employed for chemical engineering applications. However, there is no systematic assessment of OpenFOAM’s numerical accuracy and parallel performance for chemically reacting flows. For the first time, this work provides a direct comparison between OpenFOAM’s built-in flow solvers as well as its reacting flow extension EBIdnsFoam with four other, well established high-fidelity combustion codes. Quantification of OpenFOAM’s numerical accuracy is achieved with a benchmark suite that has recently been established by Abdelsamie et al. (Comput Fluids 223:104935, 2021. https://doi.org/10.1016/j.compfluid.2021.104935) for combustion codes. Fourth-order convergence can be achieved with OpenFOAM’s own cubic interpolation scheme and excellent agreement with other high-fidelity codes is presented for incompressible flows as well as more complex cases including heat conduction and molecular diffusion in multi-component mixtures. In terms of computational performance, the simulation of incompressible non-reacting flows with OpenFOAM is slower than the other codes, but similar performance is achieved for reacting flows with excellent parallel scalability. For the benchmark case of hydrogen flames interacting with a Taylor–Green vortex, differences between low-Mach and compressible solvers are identified which highlight the need for more investigations into reliable benchmarks for reacting flow solvers. The results from this work provide the first contribution of a fully implicit compressible combustion solver to the benchmark suite and are thus valuable to the combustion community. The OpenFOAM cases are publicly available and serve as guide for achieving the highest numerical accuracy as well as a basis for future developments.

Block size, parallelism and predictive performance: finding the sweet spot in distributed learning

As distributed and multi-organization Machine Learning emerges, new challenges must be solved, such as diverse and low-quality data or real-time delivery. In this paper, we use a distributed learning environment to analyze the relationship between block size, parallelism, and predictor quality. Specifically, the goal is to find the optimum block size and the best heuristic to create distributed Ensembles. We evaluated three different heuristics and five block sizes on four publicly available datasets. Results show that using fewer but better base models matches or outperforms a standard Random Forest, and that 32 MB is the best block size.

Synchronous Deterministic Parallel Programming for Multi-Cores with ForeC

Embedded real-time systems are tightly integrated with their physical environment. Their correctness depends both on the outputs and timeliness of their computations. The increasing use of multi-core processors in such systems is pushing embedded programmers to be parallel programming experts. However, parallel programming is challenging because of the skills, experiences, and knowledge needed to avoid common parallel programming traps and pitfalls. This article proposes the ForeC synchronous multi-threaded programming language for the deterministic, parallel, and reactive programming of embedded multi-cores. The synchronous semantics of ForeC is designed to greatly simplify the understanding and debugging of parallel programs. ForeC ensures that ForeC programs can be compiled efficiently for parallel execution and be amenable to static timing analysis. ForeC’s main innovation is its shared variable semantics that provides thread isolation and deterministic thread communication. All ForeC programs are correct by construction and deadlock free because no non-deterministic constructs are needed. We have benchmarked our ForeC compiler with several medium-sized programs (e.g., a 2.274-line ForeC program with up to 26 threads and distributed on up to 10 cores, which was based on a 2.155-line non-multi-threaded C program). These benchmark programs show that ForeC can achieve better parallel performance than Esterel, a widely used imperative synchronous language for concurrent safety-critical systems, and is competitive in performance to OpenMP, a popular desktop solution for parallel programming (which implements classical multi-threading, hence is intrinsically non-deterministic). We also demonstrate that the worst-case execution time of ForeC programs can be estimated to a high degree of precision.

Optimizing massively parallel sparse matrix computing on ARM many-core processor

Sparse matrix multiplication is ubiquitous in many applications such as graph processing and numerical simulation. In recent years, numerous efficient sparse matrix multiplication algorithms and computational libraries have been proposed. However, most of them are oriented to x86 or GPU platforms, while the optimization on ARM many-core platforms has not been well investigated. Our experiments show that existing sparse matrix multiplication libraries for ARM many-core CPU cannot achieve expected parallel performance. Compared with traditional multi-core CPU, ARM many-core CPU has far more cores and often adopts NUMA techniques to scale the memory bandwidth. Its parallel efficiency tends to be restricted by NUMA configuration, memory bandwidth cache contention, etc.In this paper, we propose optimized implementations for sparse matrix computing on ARM many-core CPU. We propose various optimization techniques for several routines of sparse matrix multiplication to ensure coalesced access of matrix elements in the memory. In detail, the optimization techniques include a fine-tuned CSR-based format for ARM architecture, co-optimization of Gustavson’s algorithm with hierarchical cache and dense array strategy to mitigate performance loss caused by handling compressed storage formats. We exploit the coarse-grained NUMA-aware strategy for inter-node parallelism and the fine-grained cache-aware strategy for intra-node parallelism to improve the parallel efficiency of sparse matrix multiplication. The evaluation shows that our implementation consistently outperforms the existing library on ARM many-core processor.

A modified combined active-set Newton method for solving phase-field fracture into the monolithic limit

In this work, we examine a numerical phase-field fracture framework in which the crack irreversibility constraint is treated with a primal–dual active set method and a linearization is used in the degradation function to enhance the numerical stability. The first goal is to carefully derive from a complementarity system our primal–dual active set formulation, which has been used in the literature in numerous studies, but for phase-field fracture without its detailed mathematical derivation yet. Based on the latter, we formulate a modified combined active-set Newton approach that significantly reduces the computational cost in comparison to comparable prior algorithms for quasi-monolithic settings. For many practical problems, Newton converges fast, but active set needs many iterations, for which three different efficiency improvements are suggested in this paper. Afterwards, we design an iteration on the linearization in order to iterate the problem to the monolithic limit. Our new algorithms are implemented in the programming framework pfm-cracks from Heister and Wick (2020). In the numerical examples, we conduct performance studies and investigate efficiency enhancements. The main emphasis is on the cost complexity by keeping the accuracy of numerical solutions and goal functionals. Our algorithmic suggestions are substantiated with the help of several benchmarks in two and three spatial dimensions. Therein, predictor–corrector adaptivity and parallel performance studies are explored as well.

Multirate partitioned Runge–Kutta methods for coupled Navier–Stokes equations

Earth system models are complex integrated models of atmosphere, ocean, sea ice, and land surface. Coupling the components can be a significant challenge due to the difference in physics, temporal, and spatial scales. This study explores multirate partitioned Runge–Kutta methods for the fluid–fluid interaction problem and demonstrates its parallel performance by using the PETSc library. We consider compressible Navier–Stokes equations with gravity coupled through a rigid-lid interface. Our large-scale numerical experiments reveal that multirate partitioned Runge–Kutta coupling schemes (1) can conserve total mass; (2) have second-order accuracy in time; and (3) provide favorable strong- and weak-scaling performance on modern computing architectures. We also show that the speedup factors of multirate partitioned Runge–Kutta methods match theoretical expectations over their base (single-rate) method.

A Distributed-GPU Deep Reinforcement Learning System for Solving Large Graph Optimization Problems

Graph optimization problems (such as minimum vertex cover, maximum cut, traveling salesman problems) appear in many fields including social sciences, power systems, chemistry, and bioinformatics. Recently, deep reinforcement learning (DRL) has shown success in automatically learning good heuristics to solve graph optimization problems. However, the existing RL systems either do not support graph RL environments or do not support multiple or many GPUs in a distributed setting. This has compromised the ability of reinforcement learning in solving large-scale graph optimization problems due to lack of parallelization and high scalability. To address the challenges of parallelization and scalability, we develop RL4GO , a high-performance distributed-GPU DRL framework for solving graph optimization problems. RL4GO focuses on a class of computationally demanding RL problems, where both the RL environment and policy model are highly computation intensive. Traditional reinforcement learning systems often assume either the RL environment is of low time complexity or the policy model is small. In this work, we distribute large-scale graphs across distributed GPUs and use the spatial parallelism and data parallelism to achieve scalable performance. We compare and analyze the performance of the spatial parallelism and data parallelism and show their differences. To support graph neural network (GNN) layers that take as input data samples partitioned across distributed GPUs, we design parallel mathematical kernels to perform operations on distributed 3D sparse and 3D dense tensors. To handle costly RL environments, we design a parallel graph environment to scale up all RL-environment-related operations. By combining the scalable GNN layers with the scalable RL environment, we are able to develop high-performance RL4GO training and inference algorithms in parallel. Furthermore, we propose two optimization techniques—replay buffer on-the-fly graph generation and adaptive multiple-node selection—to minimize the spatial cost and accelerate reinforcement learning. This work also conducts in-depth analyses of parallel efficiency and memory cost and shows that the designed RL4GO algorithms are scalable on numerous distributed GPUs. Evaluations on large-scale graphs show that (1) RL4GO training and inference can achieve good parallel efficiency on 192 GPUs, (2) its training time can be 18 times faster than the state-of-the-art Gorila distributed RL framework [ 34 ], and (3) its inference performance achieves a 26 times improvement over Gorila.

Reconfigurable dipole receive array for dynamic parallel imaging at ultra-high magnetic field.

To extend the concept of 3D dynamic parallel imaging, we developed a prototype of an electronically reconfigurable dipole array that provides sensitivity alteration along the dipole length. We developed a radiofrequency array coil consisting of eight reconfigurable elevated-end dipole antennas. The receive sensitivity profile of each dipole can be electronically shifted toward one or the other end by electrical shortening or lengthening the dipole arms using positive-intrinsic-negative-diode lump-element switching units. Based on the results of electromagnetic simulations, we built the prototype and tested it at 9.4 T on phantom and healthy volunteer. A modified 3D SENSE reconstruction was used, and geometry factor (g-factor) calculations were performed to assess the new array coil. Electromagnetic simulations showed that the new array coil was capable of alteration of its receive sensitivity profile along the dipole length. Electromagnetic and g-factor simulations showed closely agreeing predictions when compared to the measurements. The new dynamically reconfigurable dipole array provided significant improvement in geometry factor compared to static dipoles. We obtained up to 220% improvement for 3 × 2 (Ry × Rz ) acceleration compared to the static configuration case in terms of maximum g-factor and up to 54% in terms of mean g-factor for the same acceleration. We presented an 8-element prototype of a novel electronically reconfigurable dipole receive array that permits rapid sensitivity modulations along the dipole axes. Applying dynamic sensitivity modulation during image acquisition emulates two virtual rows of receive elements along the z-direction, and therefore improves parallel imaging performance for 3D acquisitions.

Double cross magnetic wall decoupling for quadrature transceiver RF array coils using common-mode differential-mode resonators

In contrast to linearly polarized RF coil arrays, quadrature transceiver coil arrays are capable of improving signal-to-noise ratio (SNR), spatial resolution, and parallel imaging performance. Owing to a reduced excitation power, a low specific absorption rate can also be obtained using quadrature RF coils. However, due to the complex nature of their structure and their electromagnetic properties, it is challenging to achieve sufficient electromagnetic decoupling while designing multichannel quadrature RF coil arrays, particularly in ultra-high fields. In this work, we proposed a double-cross magnetic wall decoupling for quadrature transceiver RF arrays and implemented the decoupling method on common-mode differential mode quadrature (CMDM) quadrature transceiver arrays at an ultrahigh field of 7 T. The proposed magnetic decoupling wall, comprised of two intrinsically decoupled loops, is used to reduce the mutual coupling between all the multi-mode currents present in the quadrature CMDM array. The decoupling network has no physical connection with the CMDMs’ resonators, which provides less design constraint over size-adjustable RF arrays. To validate the feasibility of the proposed cross-magnetic decoupling wall, systematic studies on the decoupling performance based on the impedance of two intrinsic loops are numerically performed. A pair of quadrature transceiver CMDMs is constructed along with the proposed decoupling network, and their scattering matrix is characterized using a network analyzer. The measured results indicate that all the current modes from coupling are simultaneously suppressed using the proposed cross-magnetic wall. Moreover, field distribution and local specific absorption rate (SAR) are numerically obtained for a well-decoupled 8-channel quadrature knee-coil array.

Catalytic Activity and Stability of Non-Platinum Group Metal Oxides for the Oxygen Evolution Reaction in Anion Exchange Membrane Electrolyzers

The activities and stabilities of non-platinum group metals (PGMs) in the forms of monometallic (Mn2O3, Fe2O3, Co3O4, NiO) and bimetallic (NiFe2O4, CoNiO2) oxides were assessed for the oxygen evolution reaction (OER) in alkaline media and compared with IrO2. Both half-cell, rotating disc electrode (RDE) apparatus and single-cell, membrane electrode assemblies (MEA) were used to study kinetic and device-level performance in parallel and to provide insights into the use of these materials in anion exchange membrane (AEM) electrolyzers. Normalization of RDE results by geometric and physical surface areas, double layer capacitance, and metal content probed differences in physically vs electrochemically accessible surface areas and ensured reported trends were independent of the normalization method. The results showed that: (i) Ni- and Co- containing materials met or exceeded IrO2 performance in both RDE and MEA testing, (ii) Co3O4 deactivated over time-on-stream (1.8 V for 13.5 h) due to oxide and, relatedly, particle growth, (iii) NiFe2O4 increased in activity over time-on-stream due to dissolution of Fe and an increased Ni/Fe ratio, and (iv) reduction of catalyst layer resistance is an avenue to further increase device-level performance. These results demonstrated the clear viability for non-PGMs to be used as anode catalysts in AEM devices.

Reinforcement Learning for Load-Balanced Parallel Particle Tracing.

We explore an online reinforcement learning (RL) paradigm to dynamically optimize parallel particle tracing performance in distributed-memory systems. Our method combines three novel components: (1) a work donation algorithm, (2) a high-order workload estimation model, and (3) a communication cost model. First, we design an RL-based work donation algorithm. Our algorithm monitors workloads of processes and creates RL agents to donate data blocks and particles from high-workload processes to low-workload processes to minimize program execution time. The agents learn the donation strategy on the fly based on reward and cost functions designed to consider processes' workload changes and data transfer costs of donation actions. Second, we propose a workload estimation model, helping RL agents estimate the workload distribution of processes in future computations. Third, we design a communication cost model that considers both block and particle data exchange costs, helping RL agents make effective decisions with minimized communication costs. We demonstrate that our algorithm adapts to different flow behaviors in large-scale fluid dynamics, ocean, and weather simulation data. Our algorithm improves parallel particle tracing performance in terms of parallel efficiency, load balance, and costs of I/O and communication for evaluations with up to 16,384 processors.

CircSSNN: circRNA-binding site prediction via sequence self-attention neural networks with pre-normalization

BackgroundCircular RNAs (circRNAs) play a significant role in some diseases by acting as transcription templates. Therefore, analyzing the interaction mechanism between circRNA and RNA-binding proteins (RBPs) has far-reaching implications for the prevention and treatment of diseases. Existing models for circRNA-RBP identification usually adopt convolution neural network (CNN), recurrent neural network (RNN), or their variants as feature extractors. Most of them have drawbacks such as poor parallelism, insufficient stability, and inability to capture long-term dependencies.MethodsIn this paper, we propose a new method completely using the self-attention mechanism to capture deep semantic features of RNA sequences. On this basis, we construct a CircSSNN model for the cirRNA-RBP identification. The proposed model constructs a feature scheme by fusing circRNA sequence representations with statistical distributions, static local contexts, and dynamic global contexts. With a stable and efficient network architecture, the distance between any two positions in a sequence is reduced to a constant, so CircSSNN can quickly capture the long-term dependencies and extract the deep semantic features.ResultsExperiments on 37 circRNA datasets show that the proposed model has overall advantages in stability, parallelism, and prediction performance. Keeping the network structure and hyperparameters unchanged, we directly apply the CircSSNN to linRNA datasets. The favorable results show that CircSSNN can be transformed simply and efficiently without task-oriented tuning.ConclusionsIn conclusion, CircSSNN can serve as an appealing circRNA-RBP identification tool with good identification performance, excellent scalability, and wide application scope without the need for task-oriented fine-tuning of parameters, which is expected to reduce the professional threshold required for hyperparameter tuning in bioinformatics analysis.

Parallel simulation via SPPARKS of on-lattice kinetic and Metropolis Monte Carlo models for materials processing

SPPARKS is an open-source parallel simulation code for developing and running various kinds of on-lattice Monte Carlo models at the atomic or meso scales. It can be used to study the properties of solid-state materials as well as model their dynamic evolution during processing. The modular nature of the code allows new models and diagnostic computations to be added without modification to its core functionality, including its parallel algorithms. A variety of models for microstructural evolution (grain growth), solid-state diffusion, thin film deposition, and additive manufacturing (AM) processes are included in the code. SPPARKS can also be used to implement grid-based algorithms such as phase field or cellular automata models, to run either in tandem with a Monte Carlo method or independently. For very large systems such as AM applications, the Stitch I/O library is included, which enables only a small portion of a huge system to be resident in memory. In this paper we describe SPPARKS and its parallel algorithms and performance, explain how new Monte Carlo models can be added, and highlight a variety of applications which have been developed within the code.

Experimental evaluation of the performance of Gpipe parallelism

Pipeline parallelism is the newly proposed model parallelism paradigm for efficiently training giant size Deep Neural Network (DNN) models across multiple accelerators. Gpipe, as a popular pipeline parallelism scheme, has been integrated into the PyTorch framework. Training a model with Gpipe involves choosing a large number of parameters, e.g. determining a DNN model partitioning scheme for the given number of accelerators, selecting the number of GPUs for the training. Therefore, it is crucial to investigate the effects of different Gpipe configurations on the model training performance and assess the scenarios that are suitable for Gpipe. This paper presents a systematic evaluation of Gpipe performance under various settings, including different DNN models, GPU types, GPU numbers, datasets, and model partition strategies. The experiments show several counterintuitive results, i.e. training a DNN model without using Gpipe performs better than using Gpipe, and utilising more GPUs does not guarantee a better performance and sometimes using less GPUs is better while training with Gpipe. Moreover, the test results also show that the GPU type, model size, dataset size, and the DNN model partition scheme clearly influence the training speed. Based on the observation of the evaluation results, the paper proposes a theoretical model to estimate the performance gain ratio while using Gpipe under different setups.

Linguistic illusions and misconceptions: The role of language variation in language development across three varieties of American English

Prior research on the linguistic abilities of Southern English- (SE) and African-American English-speaking children (SAAE) revealed unexpectedly high rates of risk for a language disorder (Christodoulou & Tsimpli 2021; Moland & Oetting 2021). This study examines the performance of 139 SE-, 46 SAAE-, and 35 Mainstream American English-speaking children (MAE), aged 2-13, and analyzes their performance, through twelve sections, in four key linguistic domains: syntax, semantics, pragmatics, and phonology, using a standardized assessment test. Results revealed a parallel performance across the three groups in all linguistic domains. The highest means of accuracy were noted with phonology and the lowest with semantics. Analysis of the participants’ performance by age evidenced a virtually identical performance across the three groups after age 6 or 7, but considerable variations were noted with younger children. Results from the current study contradict results from previous work showing considerably high rates of risk for a language disorder for the SAAE-speaking children, as their performance is parallel to not only that of SE-speaking children but it also the performance of MAE-speaking children. Results from the current study could help guide educational policies, especially for early education programs, as well as diagnostic assessment and rehabilitation.

5T magnetic resonance imaging: radio frequency hardware and initial brain imaging.

We aimed to demonstrate the feasibility of generating high-resolution human brain magnetic resonance imaging (MRI) at 5 Tesla (T) using a quadrature birdcage transmit/48-channel receiver coil assembly. A quadrature birdcage transmit/48-channel receiver coil assembly was designed for human brain imaging at 5T. The radio frequency (RF) coil assembly was validated by electromagnetic (EM) simulations and phantom imaging experimental studies. The simulated B1+ field inside a human head phantom and inside a human head model generated by the birdcage coils driven in circularly polarized (CP) mode at 3T, 5T and 7T was compared. Signal-to-noise ratio (SNR) maps, the inverse g-factor maps for evaluation of parallel imaging performance, anatomic images, angiography images, vessel wall images and susceptibility weighted images (SWI) were acquired using the RF coil assembly at 5T and compared to those acquired using a 32-channel head coil on a 3T MRI scanner. For the EM simulations, 5T MRI provided less RF inhomogeneity compared to that of 7T. In the phantom imaging study, the distributions of the measured B1+ field were consistent with the distributions of the simulated B1+ field. In the human brain imaging study, the average SNR value of the brain in the transversal plane at 5T was 1.6 times of that at 3T. The 48-channel head coil at 5T had higher parallel acceleration capability than the 32-channel head coil at 3T. The anatomic images at 5T also showed higher SNR than those at 3T. Improved delineation of the hippocampus, lenticulostriate arteries, and basilar arteries was observed at 5T compared to 3T. SWI with a higher resolution of 0.3 mm ×0.3 mm ×1.2 mm could be acquired at 5T, which enabled better visualization of small blood vessels compared to that at 3T. 5T MRI can provide significant SNR improvement compared to that of 3T with less RF inhomogeneity than that of 7T. The ability to obtain high quality in vivo human brain images at 5T using the quadrature birdcage transmit/48-channel receiver coil assembly has significant in clinical and scientific research applications.

Computational challenges in mixed‐dimensional beam/solid coupling

AbstractThe simulation of mixed‐dimensional couplings poses a variety of computational challenges in order to enable accurate modeling and efficient solution of problems in science, engineering and biomedicine. In this contribution, we study computational challenges and bottlenecks in the mixed‐dimensional modeling of coupled beam/solid systems and discuss possible remedies to allow large‐scale computations of applications in structural engineering. In particular, we address questions of memory consumption in geometric search operations, partitioning for computations on parallel clusters and performance of the linear solver by suitable preconditioning techniques.

Accelerating the Lagrangian Particle Tracking in Hydrologic Modeling to Continental‐Scale

AbstractUnprecedented climate change and anthropogenic activities have induced increasing ecohydrological problems, motivating the development of large‐scale hydrologic modeling for solutions. Water age/quality is as important as water quantity for understanding the terrestrial water cycle. However, scientific progress in tracking water parcels at large‐scale with high spatiotemporal resolutions is far behind that in simulating water balance/quantity owing to the lack of powerful modeling tools. EcoSLIM is a particle tracking model working with ParFlow‐CLM that couples integrated surface‐subsurface hydrology with land surface processes. Here, we demonstrate a parallel framework on distributed, multi‐Graphics Processing Unit platforms with Compute Unified Device Architecture‐Aware Message Passing Interface for accelerating EcoSLIM to continental‐scale. In tests from catchment‐, to regional‐, and then to continental‐scale using 25‐million to 1.6‐billion particles, EcoSLIM shows significant speedup and excellent parallel performance. The parallel framework is portable to atmospheric and oceanic particle tracking models, where the parallelization is inadequate, and a standard parallel framework is also absent. The parallelized EcoSLIM is a promising tool to accelerate our understanding of the terrestrial water cycle and the upscaling of subsurface hydrology to Earth System Models.