Multiple Graphics Processing Units Research Articles

Distributed hybrid quantum-classical performance prediction for hyperparameter optimization

Hyperparameter optimization (HPO) of neural networks is a computationally expensive procedure, which requires a large number of different model configurations to be trained. To reduce such costs, this work presents a distributed, hybrid workflow, that runs the training of the neural networks on multiple graphics processing units (GPUs) on a classical supercomputer, while predicting the configurations’ performance with quantum-trained support vector regression (QT-SVR) on a quantum annealer (QA). The workflow is shown to run on up to 50 GPUs and a QA at the same time, completely automating the communication between the classical and the quantum systems. The approach is evaluated extensively on several benchmarking datasets from the computer vision (CV), high-energy physics (HEP), and natural language processing (NLP) domains. Empirical results show that resource costs for performing HPO can be reduced by up to 9% when using the hybrid workflow with performance prediction, compared to using a plain HPO algorithm without performance prediction. Additionally, the workflow obtains similar and in some cases even better accuracy of the final hyperparameter configuration, when combining multiple heuristically obtained predictions from the QA, compared to using just a single classically obtained prediction. The results highlight the potential of hybrid quantum-classical machine learning algorithms. The workflow code is made available open-source to foster adoption in the community.

Multi-GPU RI-HF Energies and Analytic Gradients─Toward High-Throughput Ab Initio Molecular Dynamics.

This article presents an optimized algorithm and implementation for calculating resolution-of-the-identity Hartree-Fock (RI-HF) energies and analytic gradients using multiple graphics processing units (GPUs). The algorithm is especially designed for high throughput ab initio molecular dynamics simulations of small and medium size molecules (10-100 atoms). Key innovations of this work include the exploitation of multi-GPU parallelism and a workload balancing scheme that efficiently distributes computational tasks among GPUs. Our implementation also employs techniques for symmetry utilization, integral screening, and leveraging sparsity to optimize memory usage. Computational results show that the implementation achieves significant performance improvements, including over 3 × speedups in single GPU AIMD throughput compared to previous GPU-accelerated RI-HF and traditional HF methods. Furthermore, utilizing multiple GPUs can provide superlinear speedup when the additional aggregate GPU memory allows for the storage of decompressed three-center integrals.

Clustering one million molecular structures on GPU within seconds.

Structure clustering is a general but time-consuming work in the study of life science. Up to now, most published tools do not support the clustering analysis on graphics processing unit (GPU) with root mean square deviation metric. In this work, we specially write codes to do the work. It supports multiple threads on multiple GPUs. To show the performance, we apply the program to a 33-residue fragment in protein Pin1 WW domain mutant. The dataset contains 1,400,000 snapshots, which are extracted from an enhanced sampling simulation and distribute widely in the conformational space. Various testing results present that our program is quite efficient. Particularly, with two NVIDIA RTX4090 GPUs and single precision data type, the clustering calculation on 1 million snapshots is completed in a few seconds (including the uploading time of data from memory to GPU and neglecting the reading time from hard disk). This is hundreds of times faster than central processing unit. Our program could be a powerful tool for fast extraction of representative states of a molecule among its thousands to millions of candidate structures.

Multi-GPU 3D k-nearest neighbors computation with application to ICP, point cloud smoothing and normals computation

The k-Nearest Neighbors algorithm is a fundamental algorithm that finds applications in many fields like Machine Learning, Computer Graphics, Computer Vision, and others. The algorithm determines the closest points (d-dimensional) of a reference set R according to a query set of points Q under a specific metric (Euclidean, Mahalanobis, Manhattan, etc.). This work focuses on the utilization of multiple Graphical Processing Units for the acceleration of the k-Nearest Neighbors algorithm with large or very large sets of 3D points. With the proposed approach the space of the reference set is divided into a 3D grid which is used to facilitate the search for the nearest neighbors. The search in the grid is performed in a multiresolution manner starting from a high-resolution grid and ending up in a coarse one, thus accounting for point clouds that may have non-uniform sampling and/or outliers. Three important algorithms in reverse engineering are revisited and new multi-GPU versions are proposed based on the introduced KNN algorithm. More specifically, the new multi-GPU approach is applied to the Iterative Closest Point algorithm, to the point cloud smoothing, and to the point cloud normal vectors computation and orientation problem. A series of tests and experiments have been conducted and discussed in the paper showing the merits of the proposed multi-GPU approach.

Mother-leaf-method accelerated parallel-GPU AMR phase-field simulations of dendrite growth

Parallel computing with multiple graphics processing units (GPUs) for adaptive mesh refinement (AMR) is a powerful method that improves the computational efficiency of phase-field (PF) simulations of dendrite growth. Octree-type block-structured AMR is often used in parallel computation because of its good parallelism; however, the process of filling the buffer region with physical data from adjacent blocks constitutes a major overhead cost that reduces computational performance. In this study, the mother-leaf (ML) method, which can reduce this overhead, was applied to accelerate parallel-GPU AMR PF simulations of dendrite growth. To evaluate the computational performance of the ML method, single- and parallel-GPU AMR PF simulations of dendrite growth were performed. The results show that single- and parallel-GPU AMR computations with the ML method accelerate the PF simulations of dendrite growth approximately 1.2–1.6 times and 1.2–1.4 times, respectively, compared to the conventional method without the ML method. In large-scale dendrite-growth PF simulations, such accelerations are significant for investigating various dendrite growth phenomena.

Regional-scale fault-to-structure earthquake simulations with the EQSIM framework: Workflow maturation and computational performance on GPU-accelerated exascale platforms

Continuous advancements in scientific and engineering understanding of earthquake phenomena, combined with the associated development of representative physics-based models, is providing a foundation for high-performance, fault-to-structure earthquake simulations. However, regional-scale applications of high-performance models have been challenged by the computational requirements at the resolutions required for engineering risk assessments. The EarthQuake SIMulation (EQSIM) framework, a software application development under the US Department of Energy (DOE) Exascale Computing Project, is focused on overcoming the existing computational barriers and enabling routine regional-scale simulations at resolutions relevant to a breadth of engineered systems. This multidisciplinary software development—drawing upon expertise in geophysics, engineering, applied math and computer science—is preparing the advanced computational workflow necessary to fully exploit the DOE’s exaflop computer platforms coming online in the 2023 to 2024 timeframe. Achievement of the computational performance required for high-resolution regional models containing upward of hundreds of billions to trillions of model grid points requires numerical efficiency in every phase of a regional simulation. This includes run time start-up and regional model generation, effective distribution of the computational workload across thousands of computer nodes, efficient coupling of regional geophysics and local engineering models, and application-tailored highly efficient transfer, storage, and interrogation of very large volumes of simulation data. This article summarizes the most recent advancements and refinements incorporated in the workflow design for the EQSIM integrated fault-to-structure framework, which are based on extensive numerical testing across multiple graphics processing unit (GPU)-accelerated platforms, and demonstrates the computational performance achieved on the world’s first exaflop computer platform through representative regional-scale earthquake simulations for the San Francisco Bay Area in California, USA.

Multi-phase-field lattice Boltzmann simulations of semi-solid simple shear deformation in thin film

Semi-solid deformation during casting often results in significant solidification defects, such as segregation bands. Consequently, the development of a numerical simulation tool is crucial for accurately replicating semi-solid deformation. In our previous study, we applied a multi-phase-field lattice Boltzmann (MPF-LB) model to semi-solid deformation, facilitating seamless simulation from polycrystalline solidification to semi-solid deformation in a two-dimensional (2D) problem. This study extends the 2D MPF-LB model to a three-dimensional (3D) problem and develops a simulation method for semi-solid simple shear deformation in thin films. To enhance the efficiency of the 3D semi-solid simulation, we implemented parallel computations using multiple graphics processing units. Through a discussion of the relationships among the stress–strain curve, grain rearrangement behavior, and fluid flow, we confirmed that the developed 3D MPF-LB model successfully reproduced the characteristic phenomena of semi-solid deformation, and has high potential to investigate the nuanced mechanisms of semi-solid deformation.

Localization of overdamped bosonic modes and transport in strange metals

The strange metal phase of correlated electrons materials was described in a recent theory by a model of a Fermi surface coupled a two-dimensional quantum critical bosonic field with a spatially random Yukawa coupling. With the assumption of self-averaging randomness, similar to that in the Sachdev-Ye-Kitaev model, numerous observed properties of a strange metal were obtained for a wide range of intermediate temperatures, including the linear in temperature resistivity. The Harris criterion implies that spatial fluctuations in the local position of the critical point must dominate at lower temperatures. For an [Formula: see text]-component boson with [Formula: see text], we use multiple graphics processing units (GPUs) to compute the real frequency spectrum of the boson propagator in a self-consistent mean-field treatment of the boson self-interactions, but an exact treatment of multiple realizations of the spatial randomness from the random boson mass. We find that Landau damping from the fermions leads to the emergence of the physics of the random transverse-field Ising model at low temperatures, as has been proposed by Hoyos, Kotabage, and Vojta. This regime is controlled by localized overdamped eigenmodes of the bosonic scalar field, also has a resistivity which is nearly linear-in-temperature, and extends into a "quantum critical phase" away from the quantum critical point, as observed in several cuprates. For the [Formula: see text] Ising scalar, the mean-field treatment is not applicable, and so we use Hybrid Monte Carlo simulations running on multiple GPUs; we find a rounded transition and localization physics, with strange metal behavior in an extended region around the transition.

Iterative Reconstruction of Micro Computed Tomography Scans Using Multiple Heterogeneous GPUs.

Graphics processing units (GPUs) facilitate massive parallelism and high-capacity storage, and thus are suitable for the iterative reconstruction of ultrahigh-resolution micro computed tomography (CT) scans by on-the-fly system matrix (OTFSM) calculation using ordered subsets expectation maximization (OSEM). We propose a finite state automaton (FSA) method that facilitates iterative reconstruction using a heterogeneous multi-GPU platform through parallelizing the matrix calculations derived from a ray tracing system of ordered subsets. The FSAs perform flow control for parallel threading of the heterogeneous GPUs, which minimizes the latency of launching ordered-subsets tasks, reduces the data transfer between the main system memory and local GPU memory, and solves the memory-bound of a single GPU. In the experiments, we compared the operation efficiency of OS-MLTR for three reconstruction environments. The heterogeneous multiple GPUs with job queues for high throughput calculation speed is up to five times faster than the single GPU environment, and that speed up is nine times faster than the heterogeneous multiple GPUs with the FIFO queues of the device scheduling control. Eventually, we proposed an event-triggered FSA method for iterative reconstruction using multiple heterogeneous GPUs that solves the memory-bound issue of a single GPU at ultrahigh resolutions, and the routines of the proposed method were successfully executed on each GPU simultaneously.

High-Performance Multi-GPU Analytic RI-MP2 Energy Gradients.

This article presents a novel algorithm for the calculation of analytic energy gradients from second-order Møller-Plesset perturbation theory within the Resolution-of-the-Identity approximation (RI-MP2), which is designed to achieve high performance on clusters with multiple graphical processing units (GPUs). The algorithm uses GPUs for all major steps of the calculation, including integral generation, formation of all required intermediate tensors, solution of the Z-vector equation and gradient accumulation. The implementation in the EXtreme Scale Electronic Structure System (EXESS) software package includes a tailored, highly efficient, multistream scheduling system to hide CPU-GPU data transfer latencies and allows nodes with 8 A100 GPUs to operate at over 80% of theoretical peak floating-point performance. Comparative performance analysis shows a significant reduction in computational time relative to traditional multicore CPU-based methods, with our approach achieving up to a 95-fold speedup over the single-node performance of established software such as Q-Chem and ORCA. Additionally, we demonstrate that pairing our implementation with the molecular fragmentation framework in EXESS can drastically lower the computational scaling of RI-MP2 gradient calculations from quintic to subquadratic, enabling further substantial savings in runtime while retaining high numerical accuracy in the resulting gradients.

Multi-Device Parallel MRI Reconstruction: Efficient Partitioning for Undersampled 5D Cardiac CINE.

Cardiac CINE, a form of dynamic cardiac MRI, is indispensable in the diagnosis and treatment of heart conditions, offering detailed visualization essential for the early detection of cardiac diseases. As the demand for higher-resolution images increases, so does the volume of data requiring processing, presenting significant computational challenges that can impede the efficiency of diagnostic imaging. Our research presents an approach that takes advantage of the computational power of multiple Graphics Processing Units (GPUs) to address these challenges. GPUs are devices capable of performing large volumes of computations in a short period, and have significantly improved the cardiac MRI reconstruction process, allowing images to be produced faster. The innovation of our work resides in utilizing a multi-device system capable of processing the substantial data volumes demanded by high-resolution, five-dimensional cardiac MRI. This system surpasses the memory capacity limitations of single GPUs by partitioning large datasets into smaller, manageable segments for parallel processing, thereby preserving image integrity and accelerating reconstruction times. Utilizing OpenCL technology, our system offers adaptability and cross-platform functionality, ensuring wider applicability. The proposed multi-device approach offers an advancement in medical imaging, accelerating the reconstruction process and facilitating faster and more effective cardiac health assessment.

PARALLEL AND DISTRIBUTED COMPUTING TECHNOLOGIES FOR AUTONOMOUS VEHICLE NAVIGATION

Context. Autonomous vehicles are becoming increasingly popular, and one of the important modern challenges in their development is ensuring their effective navigation in space and movement within designated lanes. This paper examines a method of spatial orientation for vehicles using computer vision and artificial neural networks. The research focused on the navigation system of an autonomous vehicle, which incorporates the use of modern distributed and parallel computing technologies.
 Objective. The aim of this work is to enhance modern autonomous vehicle navigation algorithms through parallel training of artificial neural networks and to determine the optimal combination of technologies and nodes of devices to increase speed and enable real-time decision-making capabilities in spatial navigation for autonomous vehicles.
 Method. The research establishes that the utilization of computer vision and neural networks for road lane segmentation proves to be an effective method for spatial orientation of autonomous vehicles. For multi-core computing systems, the application of parallel programming technology, OpenMP, for neural network training on processors with varying numbers of parallel threads increases the algorithm’s execution speed. However, the use of CUDA technology for neural network training on a graphics processing unit significantly enhances prediction speeds compared to OpenMP. Additionally, the feasibility of employing PyTorch Distributed Data Parallel (DDP) technology for training the neural network across multiple graphics processing units (nodes) simultaneously was explored. This approach further improved prediction execution times compared to using a single graphics processing unit.
 Results. An algorithm for training and prediction of an artificial neural network was developed using two independent nodes, each equipped with separate graphics processing units, and their synchronization for exchanging training results after each epoch, employing PyTorch Distributed Data Parallel (DDP) technology. This approach allows for scalable computations across a higher number of resources, significantly expediting the model training process.
 Conclusions. The conducted experiments have affirmed the effectiveness of the proposed algorithm, warranting the recommendation of this research for further advancement in autonomous vehicles and enhancement of their navigational capabilities. Notably, the research outcomes can find applications in various domains, encompassing automotive manufacturing, logistics, and urban transportation infrastructure. The obtained results are expected to assist future researchers in understanding the most efficient hardware and software resources to employ for implementing AI-based navigation systems in autonomous vehicles. Prospects for future investigations may encompass refining the accuracy of the proposed parallel algorithm without compromising its efficiency metrics. Furthermore, there is potential for experimental exploration of the proposed algorithm in more intricate practical scenarios of diverse nature and dimensions.

AD-NEv: A Scalable Multilevel Neuroevolution Framework for Multivariate Anomaly Detection.

Anomaly detection tools and methods present a key capability in modern cyberphysical and failure prediction systems. Despite the fast-paced development in deep learning architectures for anomaly detection, model optimization for a given dataset is a cumbersome and time-consuming process. Neuroevolution could be an effective and efficient solution to this problem, as a fully automated search method for learning optimal neural networks, supporting both gradient and nongradient fine-tuning. However, existing methods mostly focus on optimizing model architectures without taking into account feature subspaces and model weights. In this work, we propose anomaly detection neuroevolution (AD-NEv)-a scalable multilevel optimized neuroevolution framework for multivariate time-series anomaly detection. The method represents a novel approach to synergically: 1) optimize feature subspaces for an ensemble model based on the bagging technique; 2) optimize the model architecture of single anomaly detection models; and 3) perform nongradient fine-tuning of network weights. An extensive experimental evaluation on widely adopted multivariate anomaly detection benchmark datasets shows that the models extracted by AD-NEv outperform well-known deep learning architectures for anomaly detection. Moreover, results show that AD-NEv can perform the whole process efficiently, presenting high scalability when multiple graphics processing units (GPUs) are available.

GPU-based transient analysis of modern grids deploying a hybrid DDM algorithm

Solving a significant number of differential-algebraic equations is essential for conducting transient stability simulations on a complex and interconnected electrical power system. Because of the increasing size and intricacy of modern grids, traditional sequential simulation methods are becoming more computationally challenging and time-consuming to use for dynamic simulations. This research aims to examine parallelism techniques from an innovative algorithm viewpoint to the hardware level by utilizing multiple graphics processing units as the main computational tool designed to simulate the dynamic behaviour of power systems on a large scale. Our approach features a novel hybrid algorithm to improve the dynamic simulations of grids with significant renewable energy integration. The algorithm employs a Schwarz-based approach in its initial decomposition phase to isolate synchronous machines and renewable sources from the grid. This separation enhances the algorithm's ability to manage these components effectively. Subsequently, each subsystem undergoes further division into subdomains in the second stage, with separate computations carried out using the Schur-complement technique. The simulations were conducted on various test systems, with a maximum of 25,000 buses, 8,000 synchronous generators, and 256 PV farms all modelled in detail. We introduce a GPU-oriented preprocessing and vectorization parallelization method for the implementation of our two-stage algorithm. This approach provides the flexibility to tailor parallelization techniques, including nested for loops and kernel pragmas, to craft a hardware solution that enhances the performance of our dynamic domain decomposition algorithm. Simulation outcomes have verified that this approach can yield a notable 7.8x acceleration in speed when executed on an NVIDIA GeForce RTX 2070 SUPER GPU.

Accelerating micromagnetic and atomistic simulations using multiple GPUs

It is shown that micromagnetic and atomistic spin dynamics simulations can use multiple graphical processing units (GPUs) not only to reduce computation time but also to allow for a larger simulation size than is possible on a single GPU. While interactions that depend on neighboring spins, such as exchange interactions, may be implemented efficiently by transferring data between GPUs using halo regions or direct memory accesses, implementing the long-range demagnetizing interaction is the main difficulty in achieving good performance scaling, where the data transfer rate between GPUs is a significant bottleneck. A multi-GPU convolution algorithm is developed here, which relies on single-GPU FFTs executed in parallel. It is shown that even for micromagnetic simulations where the demagnetizing interaction computation time dominates, good performance scaling may be achieved, with speedup factors up to 1.8, 2.5, and 3.1, for two, three, and four GPUs, respectively. The code developed here can be used for any number of GPUs in parallel, with performance scaling strongly dependent on the inter-GPU data transfer rate and connection topology. This is further improved in micromagnetic simulations, which include a spin transport solver, obtaining speedup factors up to 1.96, 2.8, and 3.7, for two, three, and four GPUs, respectively. The best case scenario is obtained for atomistic simulations, where the demagnetizing interaction is implemented with spin-averaged cells. Using a single workstation with four GPUs, it is shown that atomistic spin dynamics simulations with up to 1 × 109 spins and atomistic Monte Carlo simulations with up to 2 × 109 spins are possible, with near-ideal performance scaling.

H-Analysis and data-parallel physics-informed neural networks

We explore the data-parallel acceleration of physics-informed machine learning (PIML) schemes, with a focus on physics-informed neural networks (PINNs) for multiple graphics processing units (GPUs) architectures. In order to develop scale-robust and high-throughput PIML models for sophisticated applications which may require a large number of training points (e.g., involving complex and high-dimensional domains, non-linear operators or multi-physics), we detail a novel protocol based on h-analysis and data-parallel acceleration through the Horovod training framework. The protocol is backed by new convergence bounds for the generalization error and the train-test gap. We show that the acceleration is straightforward to implement, does not compromise training, and proves to be highly efficient and controllable, paving the way towards generic scale-robust PIML. Extensive numerical experiments with increasing complexity illustrate its robustness and consistency, offering a wide range of possibilities for real-world simulations.

Vectorized Matrix Formulation for Sound Power Determination Through Inverse Problems and Regularization Techniques

Sound power estimation is a widely investigated research topic. Some of its state-of-the-art measurement solutions include taking velocity values on the source boundary, discrete pressure assessment on a virtual outer surface or evaluations with a sound intensity probe. The increased availability of microphone arrays proposes a new tool to obtain these results. The inverse boundary element method, the equivalent source method, the Helmholtz equation least-squares, and an approach based on the radiation resistance matrix provide vectorized formulations for estimating sound power. In this study, the results for different regularization techniques on the sound power deviation from a discrete line source are presented based on a backward reconstruction scheme from simulated sound pressure field points on four microphone arrays. Additionally, in this study, time improvement between single-frequency calculations and multi-frequency execution on multiple graphics processing units is presented. The outcome shows good accuracy and reduced computation time in graphics card implementations.

The implementation of the three-dimensional unified gas-kinetic wave-particle method on multiple graphics processing units

To further improve the efficiency of the unified gas-kinetic wave-particle (UGKWP) method in hypersonic rarefied non-equilibrium flows, particularly the particle simulation process, we presented the first application of the three-dimensional UGKWP method to multiple graphics processing unit (GPU) devices in this study. The wave and particle evolution components of the method are addressed using cell and particle paralleling strategies, respectively, enabling the primary loop of the GPU-based UPKWP (GPU-UGKWP) to be executed entirely by the compute unified device architecture threads on GPU devices. Concurrently, communication issues between central processing unit (CPU) nodes are resolved by employing the message passing interface model. Additionally, we introduce a tailored memory management scheme for the GPU-UGKWP method, facilitating efficient access to the particle array. Performance comparisons reveal that, relative to a single Intel Xeon Gold 6148 CPU core, the Nvidia Tesla P100 achieves a total speedup of 34 using one GPU device, and 226 with eight GPU devices, and a single Nvidia Titan V GPU device attains a speedup of 62. The speedup outcomes on multiple CPU cores and GPU devices demonstrate that the GPU-based algorithm is better suited for computationally demanding tasks, particularly in particle-dominated simulations. As evidenced by the reduced calculation time for a hypersonic technology vehicle simulation performed on the P100 cluster, GPU devices significantly outperform their CPU counterparts.

A Multi-GPU Aggregation-Based AMG Preconditioner for Iterative Linear Solvers

We present and release in open source format a sparse linear solver which efficiently exploits heterogeneous parallel computers. The solver can be easily integrated into scientific applications that need to solve large and sparse linear systems on modern parallel computers made of hybrid nodes hosting Nvidia Graphics Processing Unit (GPU) accelerators. The work extends previous efforts of some of the authors in the exploitation of a single GPU accelerator and proposes an implementation, based on the hybrid MPI-CUDA software environment, of a Krylov-type linear solver relying on an efficient Algebraic MultiGrid (AMG) preconditioner already available in the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">BootCMatchG</i> library. Our design for the hybrid implementation has been driven by the best practices for minimizing data communication overhead when multiple GPUs are employed, yet preserving the efficiency of the GPU kernels. Strong and weak scalability results of the new version of the library on well-known benchmark test cases are discussed. Comparisons with the Nvidia AmgX solution show a speedup, in the solve phase, up to 2.0x.

Adopting GPU computing to support DL-based Earth science applications

ABSTRACT With the advancement of Artificial Intelligence (AI) technologies and accumulation of big Earth data, Deep Learning (DL) has become an important method to discover patterns and understand Earth science processes in the past several years. While successful in many Earth science areas, AI/DL applications are often challenging for computing devices. In recent years, Graphics Processing Unit (GPU) devices have been leveraged to speed up AI/DL applications, yet computational performance still poses a major barrier for DL-based Earth science applications. To address these computational challenges, we selected five existing sample Earth science AI applications, revised the DL-based models/algorithms, and tested the performance of multiple GPU computing platforms to support the applications. Application software packages, performance comparisons across different platforms, along with other results, are summarized. This article can help understand how various AI/ML Earth science applications can be supported by GPU computing and help researchers in the Earth science domain better adopt GPU computing (such as supermicro, GPU clusters, and cloud computing-based) for their AI/ML applications, and to optimize their science applications to better leverage the computing device.