Graphics Processing Unit Cluster Research Articles

Large-scale simulations on multiple Graphics Processing Units (GPUs) for the direct simulation Monte Carlo method

In this study, the application of the two-dimensional direct simulation Monte Carlo (DSMC) method using an MPI-CUDA parallelization paradigm on Graphics Processing Units (GPUs) clusters is presented. An all-device (i.e. GPU) computational approach is adopted where the entire computation is performed on the GPU device, leaving the CPU idle during all stages of the computation, including particle moving, indexing, particle collisions and state sampling. Communication between the GPU and host is only performed to enable multiple-GPU computation. Results show that the computational expense can be reduced by 15 and 185 times when using a single GPU and 16 GPUs respectively when compared to a single core of an Intel Xeon X5670 CPU. The demonstrated parallel efficiency is 75% when using 16 GPUs as compared to a single GPU for simulations using 30 million simulated particles. Finally, several very large-scale simulations in the near-continuum regime are employed to demonstrate the excellent capability of the current parallel DSMC method.

Hierarchical fractional-step approximations and parallel kinetic Monte Carlo algorithms

We present a mathematical framework for constructing and analyzing parallel algorithms for lattice kinetic Monte Carlo (KMC) simulations. The resulting algorithms have the capacity to simulate a wide range of spatio-temporal scales in spatially distributed, non-equilibrium physiochemical processes with complex chemistry and transport micro-mechanisms. Rather than focusing on constructing exactly the stochastic trajectories, our approach relies on approximating the evolution of observables, such as density, coverage, correlations and so on. More specifically, we develop a spatial domain decomposition of the Markov operator (generator) that describes the evolution of all observables according to the kinetic Monte Carlo algorithm. This domain decomposition corresponds to a decomposition of the Markov generator into a hierarchy of operators and can be tailored to specific hierarchical parallel architectures such as multi-core processors or clusters of Graphical Processing Units (GPUs). Based on this operator decomposition, we formulate parallel Fractional step kinetic Monte Carlo algorithms by employing the Trotter Theorem and its randomized variants; these schemes, (a) are partially asynchronous on each fractional step time-window, and (b) are characterized by their communication schedule between processors.The proposed mathematical framework allows us to rigorously justify the numerical and statistical consistency of the proposed algorithms, showing the convergence of our approximating schemes to the original serial KMC. The approach also provides a systematic evaluation of different processor communicating schedules. We carry out a detailed benchmarking of the parallel KMC schemes using available exact solutions, for example, in Ising-type systems and we demonstrate the capabilities of the method to simulate complex spatially distributed reactions at very large scales on GPUs. Finally, we discuss work load balancing between processors and propose a re-balancing scheme based on probabilistic mass transport methods.

Iterative sparse matrix–vector multiplication for accelerating the block Wiedemann algorithm over GF(2) on multi‐graphics processing unit systems

SUMMARYThe block Wiedemann (BW) algorithm is frequently used to solve sparse linear systems over GF(2). Iterative sparse matrix–vector multiplication is the most time‐consuming operation. The necessity to accelerate this step is motivated by the application of BW to very large matrices used in the linear algebra step of the number field sieve (NFS) for integer factorization. In this paper, we derive an efficient CUDA implementation of this operation by using a newly designed hybrid sparse matrix format. This leads to speedups between 4 and 8 on a single graphics processing unit (GPU) for a number of tested NFS matrices compared with an optimized multicore implementation. We further present a GPU cluster implementation of the full BW for NFS matrices. A small‐sized GPU cluster is able to outperform CPU clusters of larger size for large matrices such as the one obtained from the Kilobit special NFS factorization. Copyright © 2012 John Wiley & Sons, Ltd.

Numerical Analysis of Long Range Sound Wave Propagation in Ocean by Wave Equation Finite Difference Time Domain Method with Graphics Processing Unit

The wave equation finite difference time domain (WE-FDTD) method is applied to the analysis of the long range sound wave propagation in the deep ocean. In the WE-FDTD method, the wave equation in the cylindrical coordinate is directly discretized on the basis of the central differences. The method is then implemented on a graphics processing unit (GPU) cluster system, which consists of 32 GPUs. Assuming the axisymmetric field, two-dimensional numerical models whose region size is 1000 km × 5000 m are developed for various cell sizes (1–3 m). Some numerical demonstrations are made for sound wave propagation in the deep ocean under the assumption of the Munk profile, which is known as the sound speed profile of the mid-latitude of the Pacific Ocean. The numerical results are compared with the results obtained using the ray-tracing method. It is found that the numerical dispersion error appears strikingly in the WE-FDTD solutions when the points per wavelength are less than 20 p.p.w., while the WE-FDTD solutions show good agreement with the ray-tracing solutions in the propagation time when the points per wavelength are more than 20 p.p.w. It is confirmed that the WE-FDTD method can be applied to the analysis of long range sound wave propagation in the deep ocean with reasonable accuracy.

Numerical Analysis of Long Range Sound Wave Propagation in Ocean by Wave Equation Finite Difference Time Domain Method with Graphics Processing Unit

The wave equation finite difference time domain (WE-FDTD) method is applied to the analysis of the long range sound wave propagation in the deep ocean. In the WE-FDTD method, the wave equation in the cylindrical coordinate is directly discretized on the basis of the central differences. The method is then implemented on a graphics processing unit (GPU) cluster system, which consists of 32 GPUs. Assuming the axisymmetric field, two-dimensional numerical models whose region size is 1000 km × 5000 m are developed for various cell sizes (1–3 m). Some numerical demonstrations are made for sound wave propagation in the deep ocean under the assumption of the Munk profile, which is known as the sound speed profile of the mid-latitude of the Pacific Ocean. The numerical results are compared with the results obtained using the ray-tracing method. It is found that the numerical dispersion error appears strikingly in the WE-FDTD solutions when the points per wavelength are less than 20 p.p.w., while the WE-FDTD solutions show good agreement with the ray-tracing solutions in the propagation time when the points per wavelength are more than 20 p.p.w. It is confirmed that the WE-FDTD method can be applied to the analysis of long range sound wave propagation in the deep ocean with reasonable accuracy.

High performance transcription factor-DNA docking with GPU computing

BackgroundProtein-DNA docking is a very challenging problem in structural bioinformatics and has important implications in a number of applications, such as structure-based prediction of transcription factor binding sites and rational drug design. Protein-DNA docking is very computational demanding due to the high cost of energy calculation and the statistical nature of conformational sampling algorithms. More importantly, experiments show that the docking quality depends on the coverage of the conformational sampling space. It is therefore desirable to accelerate the computation of the docking algorithm, not only to reduce computing time, but also to improve docking quality.MethodsIn an attempt to accelerate the sampling process and to improve the docking performance, we developed a graphics processing unit (GPU)-based protein-DNA docking algorithm. The algorithm employs a potential-based energy function to describe the binding affinity of a protein-DNA pair, and integrates Monte-Carlo simulation and a simulated annealing method to search through the conformational space. Algorithmic techniques were developed to improve the computation efficiency and scalability on GPU-based high performance computing systems.ResultsThe effectiveness of our approach is tested on a non-redundant set of 75 TF-DNA complexes and a newly developed TF-DNA docking benchmark. We demonstrated that the GPU-based docking algorithm can significantly accelerate the simulation process and thereby improving the chance of finding near-native TF-DNA complex structures. This study also suggests that further improvement in protein-DNA docking research would require efforts from two integral aspects: improvement in computation efficiency and energy function design.ConclusionsWe present a high performance computing approach for improving the prediction accuracy of protein-DNA docking. The GPU-based docking algorithm accelerates the search of the conformational space and thus increases the chance of finding more near-native structures. To the best of our knowledge, this is the first ad hoc effort of applying GPU or GPU clusters to the protein-DNA docking problem.

Overlapping computation and communication of three-dimensional FDTD on a GPU cluster

Large-scale electromagnetic field simulations using the FDTD (finite-difference time-domain) method require the use of GPU (graphics processing unit) clusters. However, the communication overhead caused by slow interconnections becomes a major performance bottleneck. In this paper, as a way to remove the bottleneck, we propose the ‘kernel-split method’ and the ‘host-buffer method’ which overlap computation and communication for the FDTD simulation on the GPU cluster. The host-buffer method in particular enables overlapping without any modifications to the update-kernels that are already in use. We also present theoretical formulas to predict the overlap threshold and the total throughput for each method. By using our overlap methods with 6 GPU nodes, we demonstrate that the total performance of 3D FDTD reaches 92% of a six-fold increase, which is the upper limit that would be reached if there were no communication overhead.

Pricing derivatives on graphics processing units using Monte Carlo simulation

SUMMARYThis paper is about using the existing Monte Carlo approach for pricing European and American contracts on a state‐of‐the‐art graphics processing unit (GPU) architecture. First, we adapt on a cluster of GPUs two different suitable paradigms of parallelizing random number generators, which were developed for CPU clusters. Because in financial applications, we request results within seconds of simulation, the sufficiently large computations should be implemented on a cluster of machines. Thus, we make the European contract comparison between CPUs and GPUs using from one up to 16 nodes of a CPU/GPU cluster. We show that using GPUs for European contracts reduces the execution time by ∼ 40 and diminishes the energy consumed by ∼ 50 during the simulation. In the second set of experiments, we investigate the benefits of using GPUs’ parallelization for pricing American options that require solving an optimal stopping problem and which we implement using the Longstaff and Schwartz regression method. The speedup result obtained for American options varies between two and 10 according to the number of generated paths, the dimensions, and the time discretization. Copyright © 2012 John Wiley & Sons, Ltd.

Fast on‐site Monte Carlo tool for dose calculations in CT applications

Monte Carlo (MC) simulation is an established technique for dose calculation in diagnostic radiology. The major drawback is its high computational demand, which limits the possibility of usage in real-time applications. The aim of this study was to develop fast on-site computed tomography (CT) specific MC dose calculations by using a graphics processing unit (GPU) cluster. GPUs are powerful systems which are especially suited to problems that can be expressed as data-parallel computations. In MC simulations, each photon track is independent of the others; each launched photon can be mapped to one thread on the GPU, thousands of threads are executed in parallel in order to achieve high performance. For further acceleration, the authors considered multiple GPUs. The total computation was divided into different parts which can be calculated in parallel on multiple devices. The GPU cluster is an MC calculation server which is connected to the CT scanner and computes 3D dose distributions on-site immediately after image reconstruction. To estimate the performance gain, the authors benchmarked dose calculation times on a 2.6 GHz Intel Xeon 5430 Quad core workstation equipped with two NVIDIA GeForce GTX 285 cards. The on-site calculation concept was demonstrated for clinical and preclinical datasets on CT scanners (multislice CT, flat-detector CT, and micro-CT) with varying geometry, spectra, and filtration. To validate the GPU-based MC algorithm, the authors measured dose values on a 64-slice CT system using calibrated ionization chambers and thermoluminesence dosimeters (TLDs) which were placed inside standard cylindrical polymethyl methacrylate (PMMA) phantoms. The dose values and profiles obtained by GPU-based MC simulations were in the expected good agreement with computed tomography dose index (CTDI) measurements and reference TLD profiles with differences being less than 5%. For 10(9) photon histories simulated in a 256 × 256 × 12 voxel thorax dataset with voxel size of 1.36 × 1.36 × 3.00 mm(3), calculation times of about 70 and 24 min were necessary with single-core and multiple-core central processing unit (CPU) solutions, respectively. Using GPUs, the same MC calculations were performed in 1.27 min (single card) and 0.65 min (two cards) without a loss in quality. Simulations were thus speeded up by factors up to 55 and 36 compared to single-core and multiple-core CPU, respectively. The performance scaled nearly linearly with the number of GPUs. Tests confirmed that the proposed GPU-based MC tool can be easily adapted to different types of CT scanners and used as service providers for fast on-site dose calculations. The Monte Carlo software package provides fast on-site calculation of 3D dose distributions in the CT suite which makes it a practical tool for any type of CT-specific application.

Efficient parallel implementation of the lattice Boltzmann method on large clusters of graphic processing units

Many-core processors, such as graphic processing units (GPUs), are promising platforms for intrinsic parallel algorithms such as the lattice Boltzmann method (LBM). Although tremendous speedup has been obtained on a single GPU compared with mainstream CPUs, the performance of the LBM for multiple GPUs has not been studied extensively and systematically. In this article, we carry out LBM simulation on a GPU cluster with many nodes, each having multiple Fermi GPUs. Asynchronous execution with CUDA stream functions, OpenMP and non-blocking MPI communication are incorporated to improve efficiency. The algorithm is tested for two-dimensional Couette flow and the results are in good agreement with the analytical solution. For both the one- and two-dimensional decomposition of space, the algorithm performs well as most of the communication time is hidden. Direct numerical simulation of a two-dimensional gas-solid suspension containing more than one million solid particles and one billion gas lattice cells demonstrates the potential of this algorithm in large-scale engineering applications. The algorithm can be directly extended to the three-dimensional decomposition of space and other modeling methods including explicit grid-based methods.

Scalability of Self-organizing Maps on a GPU cluster using OpenCL and CUDA

We evaluate a novel implementation of a Self-Organizing Map (SOM) on a Graphics Processing Unit (GPU) cluster. Using various combinations of OpenCL, CUDA, and two different graphics cards, we demonstrate the scalability of the SOM implementation on one to eight GPUs. Results indicate that while the algorithm scales well with the number of training samples and the map size, the benefits from using the data-parallel approaches offered by the GPU are severely limited when combined with the Message Passing Interface (MPI) in this setting, and comparable to speedups of GPU-based implementations as compared to optimized sequential code. Speedups achieved range from 3 to 32, for various map and training data sizes. We also observed a performance penalty for the OpenCL implementation as compared to CUDA.

A Parallel Supercomputer Implementation of a Biological Inspired Neural Network and its use for Pattern Recognition

A parallel implementation of a large spiking neural network is proposed and evaluated. The neural network implements the binding by synchrony process using the Oscillatory Dynamic Link Matcher (ODLM). Scalability, speed and performance are compared for 2 implementations: Message Passing Interface (MPI) and Compute Unified Device Architecture (CUDA) running on clusters of multicore supercomputers and NVIDIA graphical processing units respectively. A global spiking list that represents at each instant the state of the neural network is described. This list indexes each neuron that fires during the current simulation time so that the influence of their spikes are simultaneously processed on all computing units. Our implementation shows a good scalability for very large networks. A complex and large spiking neural network has been implemented in parallel with success, thus paving the road towards real-life applications based on networks of spiking neurons. MPI offers a better scalability than CUDA, while the CUDA implementation on a GeForce GTX 285 gives the best cost to performance ratio. When running the neural network on the GTX 285, the processing speed is comparable to the MPI implementation on RQCHP's Mammouth parallel with 64 notes (128 cores).

Directionally unsplit hydrodynamic schemes with hybrid MPI/OpenMP/GPU parallelization in AMR

We present the implementation and performance of a class of directionally unsplit Riemann-solver-based hydrodynamic schemes on graphics processing units (GPUs). These schemes, including the MUSCL-Hancock method, a variant of the MUSCL-Hancock method, and the corner-transport-upwind method, are embedded into the adaptive-mesh-refinement (AMR) code GAMER. Furthermore, a hybrid MPI/OpenMP model is investigated, which enables the full exploitation of the computing power in a heterogeneous CPU/GPU cluster and significantly improves the overall performance. Performance benchmarks are conducted on the Dirac GPU cluster at NERSC/LBNL using up to 32 Tesla C2050 GPUs. A single GPU achieves speed-ups of 101 (25) and 84 (22) for uniform-mesh and AMR simulations, respectively, as compared with the performance using one (four) CPU core(s), and the excellent performance persists in multi-GPU tests. In addition, we make a direct comparison between GAMER and the widely adopted CPU code Athena in adiabatic hydrodynamic tests and demonstrate that, with the same accuracy, GAMER is able to achieve two orders of magnitude performance speed-up.

Implementing molecular dynamics on hybrid high performance computers – Particle–particle particle-mesh

The use of accelerators such as graphics processing units (GPUs) has become popular in scientific computing applications due to their low cost, impressive floating-point capabilities, high memory bandwidth, and low electrical power requirements. Hybrid high-performance computers, machines with nodes containing more than one type of floating-point processor (e.g. CPU and GPU), are now becoming more prevalent due to these advantages. In this paper, we present a continuation of previous work implementing algorithms for using accelerators into the LAMMPS molecular dynamics software for distributed memory parallel hybrid machines. In our previous work, we focused on acceleration for short-range models with an approach intended to harness the processing power of both the accelerator and (multi-core) CPUs. To augment the existing implementations, we present an efficient implementation of long-range electrostatic force calculation for molecular dynamics. Specifically, we present an implementation of the particle–particle particle-mesh method based on the work by Harvey and De Fabritiis. We present benchmark results on the Keeneland InfiniBand GPU cluster. We provide a performance comparison of the same kernels compiled with both CUDA and OpenCL. We discuss limitations to parallel efficiency and future directions for improving performance on hybrid or heterogeneous computers.

Fast seismic modeling and reverse time migration on a graphics processing unit cluster

SUMMARYWe designed a fast parallel simulator that solves the acoustic wave equation on a graphics processing unit (GPU) cluster. Solving the acoustic wave equation in an oil exploration industrial context aims at speeding up seismic modeling and reverse time migration (RTM). We considered a finite difference approach on a regular mesh, in both two‐dimensional and three‐dimensional cases. The acoustic wave equation is solved in a constant density or a variable density domain. All the computations were carried out in single precision (both in the CPU reference implementation and in the GPU implementation), because double precision was not required in our context. We used Compute Unified Device Architecture to take advantage of the GPU computational power. We studied different implementations and their impact on the application performance. The described application handles all the steps of seismic modeling and RTM and is used to solve real‐world problems in an industrial production context. We obtained a speedup of 16 for RTM and up to 43 for the modeling application over a sequential code running on general‐purpose CPUs. A CPU rack versus a GPU rack comparison was described and showed a 4.3 speedup. Copyright © 2011 John Wiley & Sons, Ltd.

Patient-specific modelling of pulmonary airflow using GPU cluster for the application in medical practice

In this paper, we propose a novel patient-specific method of modelling pulmonary airflow using graphics processing unit (GPU) computation that can be applied in medical practice. To overcome the barriers imposed by computation speed, installation price and footprint to the application of computational fluid dynamics, we focused on GPU computation and the lattice Boltzmann method (LBM). The GPU computation and LBM are compatible due to the characteristics of the GPU. As the optimisation of data access is essential for the performance of the GPU computation, we developed an adaptive meshing method, in which an airway model is covered by isotropic subdomains consisting of a uniform Cartesian mesh. We found that 43 size subdomains gave the best performance. The code was also tested on a small GPU cluster to confirm its performance and applicability, as the price and footprint are reasonable for medical applications.

Acceleration of 2-D Compressible Flow Solvers with Graphics Processing Unit Clusters

We demonstrate the first use of graphics processing unit clusters for engineering-level compressible flow computations with two separately developed solvers. Each solver adopts the finite-volume approach on multiblock, structured, nonuniform, quadrilateral meshes. We use a second-order accurate integration scheme, with a second-order accurate dual-time integration, using steady-flow acceleration techniques such as local time-stepping and nonlinear multigrid. The first solver is based on an existing multiblock Fortran code, MBFLO, which is augmented with graphics processing unit acceleration for the laminar and unsteady flow subroutines. The second solver is built from scratch for graphics processing unit acceleration, but is limited to the steady Euler solutions over simple geometries. The Euler solver is optimized by reducing the memory bandwidth, recomputing intermediate values, and combining functions to increase producer-consumer locality.These two approaches are used to highlight trade-offs between development effort and performance when accelerating an application with graphics processing units. On test cases with up to 6.4 million grid points, our graphics processing unit solvers outperform their central processing unit counterparts by up to 20×. In our graphics processing unit strong scalability test, 32 graphics processing units scale up to 22× over a single graphics processing unit, which is 16× faster than 32 central processing unit cores, or 500× faster than a single Central Processing Unit core. It is confirmed that graphics processing unit clusters are effective for engineering computational fluid dynamics applications if their order-of-magnitude increase in performance is paired with large problem sizes to amortize the cost of communication.

N-body simulation for self-gravitating collisional systems with a new SIMD instruction set extension to the x86 architecture, Advanced Vector eXtensions

We present a high-performance N-body code for self-gravitating collisional systems accelerated with the aid of a new SIMD instruction set extension of the x86 architecture: Advanced Vector eXtensions (AVX), an enhanced version of the Streaming SIMD Extensions (SSE). With one processor core of Intel Core i7-2600 processor (8 MB cache and 3.40 GHz) based on Sandy Bridge micro-architecture, we implemented a fourth-order Hermite scheme with individual timestep scheme ( Makino and Aarseth, 1992), and achieved the performance of ∼20 giga floating point number operations per second (GFLOPS) for double-precision accuracy, which is two times and five times higher than that of the previously developed code implemented with the SSE instructions ( Nitadori et al., 2006b), and that of a code implemented without any explicit use of SIMD instructions with the same processor core, respectively. We have parallelized the code by using so-called NINJA scheme ( Nitadori et al., 2006a), and achieved ∼90 GFLOPS for a system containing more than N = 8192 particles with 8 MPI processes on four cores. We expect to achieve about 10 tera FLOPS (TFLOPS) for a self-gravitating collisional system with N ∼ 10 5 on massively parallel systems with at most 800 cores with Sandy Bridge micro-architecture. This performance will be comparable to that of Graphic Processing Unit (GPU) cluster systems, such as the one with about 200 Tesla C1070 GPUs ( Spurzem et al., 2010). This paper offers an alternative to collisional N-body simulations with GRAPEs and GPUs.

Graphics Processing Unit-Accelerated Boundary Element Method and Vortex Particle Method

Vortex particle methods, when combined with boundary element methods, become a powerful tool for the simulation of internal or external vortex-dominated flows. To speed up the simulations substantially, algorithms are presented herein for the implementation of a parallel adaptive multipole-accelerated treecode for evaluating the velocity field induced by vortex particles on a cluster of graphics processing units, as well as the implementation of a graphics processing unit-accelerated treecode for evaluating the velocity field induced by vortex and source boundary panels. The graphics processing unit-accelerated treecode for vortex particles outperforms its eight-core central processing unit counterpart by a factor of 43. Furthermore, the parallel efficiency of the treecode on a 32-node cluster of dual-graphics processingunitsis80%for100millionparticles.Theboundaryelementmethodtreecodecan solveforunknownsource,vortex,orcombinedstrengthsovertriangularsurfacemeshesusing all available central processing unit cores and graphics processing units. Problems with up to 1.4 million unknowns can now be solved on a single commodity desktop computer in 1min, and at that size the hybrid central processing unit/graphics processing unit outperforms a quad-core central processing unit alone by 22.5 times. The method is applied to simulations of the early stages of impulsively started flow over a sphere at Reynolds numbers 500, 1000, 2000, and 4000 based on the diameter and freestream velocity.

Monte Carlo–based fluorescence molecular tomography reconstruction method accelerated by a cluster of graphic processing units

High-speed fluorescence molecular tomography (FMT) reconstruction for 3-D heterogeneous media is still one of the most challenging problems in diffusive optical fluorescence imaging. In this paper, we propose a fast FMT reconstruction method that is based on Monte Carlo (MC) simulation and accelerated by a cluster of graphics processing units (GPUs). Based on the Message Passing Interface standard, we modified the MC code for fast FMT reconstruction, and different Green's functions representing the flux distribution in media are calculated simultaneously by different GPUs in the cluster. A load-balancing method was also developed to increase the computational efficiency. By applying the Fréchet derivative, a Jacobian matrix is formed to reconstruct the distribution of the fluorochromes using the calculated Green's functions. Phantom experiments have shown that only 10 min are required to get reconstruction results with a cluster of 6 GPUs, rather than 6 h with a cluster of multiple dual opteron CPU nodes. Because of the advantages of high accuracy and suitability for 3-D heterogeneity media with refractive-index-unmatched boundaries from the MC simulation, the GPU cluster-accelerated method provides a reliable approach to high-speed reconstruction for FMT imaging.