Hybrid CPU-GPU Research Articles

Cosmic Dawn II (CoDa II): a new radiation-hydrodynamics simulation of the self-consistent coupling of galaxy formation and reionization

ABSTRACT Cosmic Dawn II (CoDa II) is a new, fully coupled radiation-hydrodynamics simulation of cosmic reionization and galaxy formation and their mutual impact, to redshift z < 6. With 40963 particles and cells in a 94 Mpc box, it is large enough to model global reionization and its feedback on galaxy formation while resolving all haloes above 108 M⊙. Using the same hybrid CPU–GPU code RAMSES–CUDATON as CoDa I in Ocvirk et al. (2016), CoDa II modified and re-calibrated the subgrid star formation algorithm, making reionization end earlier, at z ≳ 6, thereby better matching the observations of intergalactic Lyman α opacity from quasar spectra and electron-scattering optical depth from cosmic microwave background fluctuations. CoDa II predicts a UV continuum luminosity function in good agreement with observations of high-z galaxies, especially at z = 6. As in CoDa I, reionization feedback suppresses star formation in haloes below ∼2 × 109 M⊙, though suppression here is less severe, a possible consequence of modifying the star formation algorithm. Suppression is environment dependent, occurring earlier (later) in overdense (underdense) regions, in response to their local reionization times. Using a constrained realization of lambda cold dark matter constructed from galaxy survey data to reproduce the large-scale structure and major objects of the present-day Local Universe, CoDa II serves to model both global and local reionization. In CoDa II, the Milky Way and M31 appear as individual islands of reionization, i.e. they were not reionized by the progenitor of the Virgo cluster, or by nearby groups, or by each other.

Accelerating Spatial Cross-Matching on CPU-GPU Hybrid Platform With CUDA and OpenACC.

Spatial cross-matching operation over geospatial polygonal datasets is a highly compute-intensive yet an essential task to a wide array of real-world applications. At the same time, modern computing systems are typically equipped with multiple processing units capable of task parallelization and optimization at various levels. This mandates for the exploration of novel strategies in the geospatial domain focusing on efficient utilization of computing resources, such as CPUs and GPUs. In this paper, we present a CPU-GPU hybrid platform to accelerate the cross-matching operation of geospatial datasets. We propose a pipeline of geospatial subtasks that are dynamically scheduled to be executed on either CPU or GPU. To accommodate geospatial datasets processing on GPU using pixelization approach, we convert the floating point-valued vertices into integer-valued vertices with an adaptive scaling factor as a function of the area of minimum bounding box. We present a comparative analysis of GPU enabled cross-matching algorithm implementation in CUDA and OpenACC accelerated C++. We test our implementations over Natural Earth Data and our results indicate that although CUDA based implementations provide better performance, OpenACC accelerated implementations are more portable and extendable while still providing considerable performance gain as compared to CPU. We also investigate the effects of input data size on the IO / computation ratio and note that a larger dataset compensates for IO overheads associated with GPU computations. Finally, we demonstrate that an efficient cross-matching comparison can be achieved with a cost-effective GPU.

Parallel reduction of four matrices to condensed form for a generalized matrix eigenvalue algorithm

The VZ algorithm proposed by Charles F. Van Loan (SIMA, 1975) attempts to solve the generalized type of matrix eigenvalue problem ACx = λBDx, where A, B ∈ Rn×m, C, D ∈ Rm×n, and m ≥ n, without forming products and inverses. Especially, this algorithm is suitable for solving the generalized singular value problem. Van Loan’s approach first reduces the matrices A, B, C, and D to a condensed form by the finite step initial reduction. The reduction finds orthogonal matrices Q, U, V, and Z, such that QAZ is upper Hessenberg, and QBV, ZTCU, and VTDU are upper triangular. In this initial reduction, A is reduced to upper Hessenberg form, while simultaneously preserving triangularity of other three matrices. This is done by Givens rotations, annihilating one by one element of A, and by generating three more rotations applied to other matrices per each annihilation. Such an algorithm is quite inefficient. In our work, we propose a blocked algorithm for the initial reduction, based on aggregated Givens rotations and matrix–matrix multiplications, which are applied in the outer loop updates. This algorithm has another level of blocking, exploited in the inner loop. Further, we also consider a variant of the algorithm in a hybrid CPU–GPU framework, where compute-intensive outer loop updates are performed on GPU, and can be overlapped with the reduction in the next step performed on CPU. On the other hand, application of a sequence of rotations in the inner loop is parallelized on CPU, with balanced operation count per thread. Since a large number of aggregated rotations are produced in every outer loop step, they are simultaneously accumulated before outer loop updates. These adjustments speed up original initial reduction considerably which is confirmed by numerical experiments, and the efficiency of the whole VZ algorithm is increased.

Helios-r2: A New Bayesian, Open-source Retrieval Model for Brown Dwarfs and Exoplanet Atmospheres

Abstract We present an improved, hybrid CPU-GPU atmospheric retrieval code, Helios-r2, which is applicable to medium-resolution emission spectra of brown dwarfs, in preparation for precision atmospheric spectroscopy in the era of the James Webb Space Telescope. The model is available as open-source code on the Exoclimes Simulation Platform. We subject Helios-r2 to a battery of tests of varying difficulty. The simplest test involves a mock retrieval on a forward model generated using the same radiative transfer technique, the same implementation of opacities, and the same chemistry model. The least trivial test involves a mock retrieval on synthetic spectra from the Sonora model grid, which uses a different radiative transfer technique, a different implementation of opacities, and a different chemistry model. A calibration factor, which is included to capture uncertainties in the brown dwarf radius, distance to the brown dwarf and flux calibration of the spectrum, may compensate, sometimes erroneously, for discrepancies in modeling choices and implementation. We analyze spectra of the benchmark brown dwarf GJ 570 D and the binary brown dwarf companions in the Epsilon Indi system. The retrieved surface gravities are consistent with previous studies and/or values inferred from dynamical masses (for Epsilon Indi Ba and Bb only). There remains no clear criterion on how to reject unphysical values of the retrieved brown dwarf radii. The inferred radii and corresponding masses should be taken with great caution. The retrieved carbon-to-oxygen ratios and metallicity depend on whether chemical equilibrium is assumed.

Towards extreme scale dissipative particle dynamics simulations using multiple GPGPUs

A multi-GPGPU development for Mesoscale Simulations using the Dissipative Particle Dynamics method is presented. This distributed GPU acceleration development is an extension of the DL_MESO package to MPI+CUDA in order to exploit the computational power of the latest NVIDIA cards on hybrid CPU–GPU architectures. Details about the extensively applicable algorithm implementation and memory coalescing data structures are presented. The key algorithms’ optimizations for the nearest-neighbour list searching of particle pairs for short range forces, exchange of data and overlapping between computation and communications are also given. We have carried out strong and weak scaling performance analyses with up to 4096 GPUs. A two phase mixture separation test case with 1.8 billion particles has been run on the Piz Daint supercomputer from the Swiss National Supercomputer Center. With CUDA aware MPI, proper GPU affinity, communication and computation overlap optimizations for multi-GPU version, the final optimization results demonstrated more than 94% efficiency for weak scaling and more than 80% efficiency for strong scaling. As far as we know, this is the first report in the literature of DPD simulations being run on this large number of GPUs. The remaining challenges and future work are also discussed at the end of the paper.

Crossmodal attentive skill learner: learning in Atari and beyond with audio–video inputs

This paper introduces the Crossmodal Attentive Skill Learner (CASL), integrated with the recently-introduced Asynchronous Advantage Option-Critic architecture [Harb et al. in When waiting is not an option: learning options with a deliberation cost. arXiv preprint arXiv:1709.04571, 2017] to enable hierarchical reinforcement learning across multiple sensory inputs. Agents trained using our approach learn to attend to their various sensory modalities (e.g., audio, video) at the appropriate moments, thereby executing actions based on multiple sensory streams without reliance on supervisory data. We demonstrate empirically that the sensory attention mechanism anticipates and identifies useful latent features, while filtering irrelevant sensor modalities during execution. Further, we provide concrete examples in which the approach not only improves performance in a single task, but accelerates transfer to new tasks. We modify the Arcade Learning Environment [Bellemare et al. in J Artif Intell Res 47:253–279, 2013] to support audio queries (ALE-audio code available at https://github.com/shayegano/Arcade-Learning-Environment), and conduct evaluations of crossmodal learning in the Atari 2600 games H.E.R.O. and Amidar. Finally, building on the recent work of Babaeizadeh et al. [in: International conference on learning representations (ICLR), 2017], we open-source a fast hybrid CPU–GPU implementation of CASL (CASL code available at https://github.com/shayegano/CASL).

A scalable multiple pairwise protein sequence alignment acceleration using hybrid CPU–GPU approach

Bioinformatics is an interdisciplinary field that applies trending techniques in information technology, mathematics, and statistics in studying large biological data. Bioinformatics involves several computational techniques such as sequence and structural alignment, data mining, macromolecular geometry, prediction of protein structure and gene finding. Protein structure and sequence analysis are vital to the understanding of cellular processes. Understanding cellular processes contributes to the development of drugs for metabolic pathways. Protein sequence alignment is concerned with identifying the similarities and the relationships among different protein structures. In this paper, we target two well-known protein sequence alignment algorithms, the Needleman–Wunsch and the Smith–Waterman algorithms. These two algorithms are computationally expensive which hinders their applicability for large data sets. Thus, we propose a hybrid parallel approach that combines the capabilities of multi-core CPUs and the power of contemporary GPUs, and significantly speeds up the execution of the target algorithms. The validity of our approach is tested on real protein sequences. Moreover, the scalability of the approach is verified on randomly generated sequences with predefined similarity levels. The results showed that the proposed hybrid approach was up to 242 times faster than the sequential approach.

Hybrid CPU-GPU Community Detection in Weighted Networks

Recently, a new trend has emerged in the field of parallel and high performance computing, the hybrid implementation using CPU-GPU modules. In such implementations, the computational load is shared between the CPU and GPU, in order to improve the computational efficiency. However, the task of sharing the computational load between the two modules is a rather difficult one, with a number of limitations being imposed. This paper extends our recent work on community detection, which is based on transforming a network of nodes into a set of threaded binary trees. In this work, we share the computational load between the two units: the CPU takes specific samples of the network communities and organizes them in the form of threaded binary trees. The GPU takes over the heavy load of reading this data and transforming it into a path-matrix. Finally, this matrix is sent back to the CPU for analysis, community detection and overlaps, as well as network information upgrades. Our simulation results show significant improvement over our previous strategy and other known community detection strategies found in the literature.

Parallel Performance Analysis and Numerical Simulation of Magnetic Nanoparticles Transportation and Blood Flow Pattern in Capillaries

A hybrid CPU-GPU approach is used to investigate the patterns of blood flow and magnetic particles numerically in a capillary under the existence of constant magnetic field. However, the blood flow is considered to be Newtonian, laminar and incompressible in the capillary and magnetic Nanoparticles are assumed as potential agent carrier being used therapeutically for the magnetic targeted drug transport in the fight against diseased cells. The magnetic field is embedded in the muscular volume and is produced by a permanent magnet. The flow field and particles transport dynamics are formulated by a mathematical model and is solved numerically. Finite element discretization gives a big sparse system of equations which needs a higher computation. Therefore, the hybrid approach deals the extensive computations in parallel, as this is a good platform which can decrease the times of solution significantly, if compared to the CPU application. This serves for highly effective search for distinctive mathematical model, along with their distinctive parameters. The impact of pressure P, the capillary radius R; the magnetic nanoparticle radius RM, the magnetic field intensity H on the blood flow along with magnetic particles is studied with regard to the inputs and model of magnetic field. The mathematical results for both the velocity of particles and blood are calculated. It is observed that magnetic field is directly proportional to the flow pattern as an increase in the former upsurge the latter. Moreover, the distance between magnet and capillary wall exerts a direct influence on velocity and is helpful to pull magnetic nanoparticles to the capillary wall. Thus, it is concluded by simulation that the magnetic parameters govern the velocity profile.

Lenstool-HPC: A High Performance Computing based mass modelling tool for cluster-scale gravitational lenses

With the upcoming generation of telescopes, cluster scale strong gravitational lenses will act as an increasingly relevant probe of cosmology and dark matter. The better resolved data produced by current and future facilities requires faster and more efficient lens modeling software. Consequently, we present Lenstool-HPC, a strong gravitational lens modeling and map generation tool based on High Performance Computing (HPC) techniques and the renowned Lenstool software. We also showcase the HPC concepts needed for astronomers to increase computation speed through massively parallel execution on supercomputers. Lenstool-HPC was developed using lens modelling algorithms with high amounts of parallelism. Each algorithm was implemented as a highly optimised CPU, GPU and Hybrid CPU-GPU version. The software was deployed and tested on the Piz Daint cluster of the Swiss National Supercomputing Centre (CSCS). Lenstool-HPC perfectly parallel lens map generation and derivative computation achieves a factor 30 speed-up using only 1 GPUs compared to Lenstool. Lenstool-HPC hybrid Lens-model fit generation tested at Hubble Space Telescope precision is scalable up to 200 CPU-GPU nodes and is faster than Lenstool using only 4 CPU-GPU nodes.

The ESCAPE project: Energy-efficient Scalable Algorithms for Weather Prediction at Exascale

Abstract. In the simulation of complex multi-scale flows arising in weather and climate modelling, one of the biggest challenges is to satisfy strict service requirements in terms of time to solution and to satisfy budgetary constraints in terms of energy to solution, without compromising the accuracy and stability of the application. These simulations require algorithms that minimise the energy footprint along with the time required to produce a solution, maintain the physically required level of accuracy, are numerically stable, and are resilient in case of hardware failure. The European Centre for Medium-Range Weather Forecasts (ECMWF) led the ESCAPE (Energy-efficient Scalable Algorithms for Weather Prediction at Exascale) project, funded by Horizon 2020 (H2020) under the FET-HPC (Future and Emerging Technologies in High Performance Computing) initiative. The goal of ESCAPE was to develop a sustainable strategy to evolve weather and climate prediction models to next-generation computing technologies. The project partners incorporate the expertise of leading European regional forecasting consortia, university research, experienced high-performance computing centres, and hardware vendors. This paper presents an overview of the ESCAPE strategy: (i) identify domain-specific key algorithmic motifs in weather prediction and climate models (which we term Weather & Climate Dwarfs), (ii) categorise them in terms of computational and communication patterns while (iii) adapting them to different hardware architectures with alternative programming models, (iv) analyse the challenges in optimising, and (v) find alternative algorithms for the same scheme. The participating weather prediction models are the following: IFS (Integrated Forecasting System); ALARO, a combination of AROME (Application de la Recherche à l'Opérationnel à Meso-Echelle) and ALADIN (Aire Limitée Adaptation Dynamique Développement International); and COSMO–EULAG, a combination of COSMO (Consortium for Small-scale Modeling) and EULAG (Eulerian and semi-Lagrangian fluid solver). For many of the weather and climate dwarfs ESCAPE provides prototype implementations on different hardware architectures (mainly Intel Skylake CPUs, NVIDIA GPUs, Intel Xeon Phi, Optalysys optical processor) with different programming models. The spectral transform dwarf represents a detailed example of the co-design cycle of an ESCAPE dwarf. The dwarf concept has proven to be extremely useful for the rapid prototyping of alternative algorithms and their interaction with hardware; e.g. the use of a domain-specific language (DSL). Manual adaptations have led to substantial accelerations of key algorithms in numerical weather prediction (NWP) but are not a general recipe for the performance portability of complex NWP models. Existing DSLs are found to require further evolution but are promising tools for achieving the latter. Measurements of energy and time to solution suggest that a future focus needs to be on exploiting the simultaneous use of all available resources in hybrid CPU–GPU arrangements.

A GPU Optimized Technique for Scalable Spiking Neural Network Simulation

Simulation studies, in general, heavily rely upon the internal variables of the system / entity in the studies. In case of simulation study of the Spiking Neural Networks (SNNs), the major internal system variables are membrane potentials of the neurons and their respective synaptic inputs which demand to be updated at a sub-millisecond resolution. It would be very apt here to note that this requires thousands of updates to simulate one second of an activity per neuron and this factor makes it imperative to have a highly scalable model to derive some inferences from the simulation. Conventionally, high performance CPUs with high degree of multi-threading were leveraged to conduct simulations and derive inferences. With the advances in the hardware, the degree of parallelism has also increased, especially the GPUs have opened a multitude of avenues to perform SNN simulations at scale. In our pervious works [1, 2, 3], we have demonstrated how GPUs can be leveraged to achieve scalability and performance by using hybrid CPU-GPU approach which have improved the performance as compared to multi-threading on high performance CPUs. In this work, we have focused on hyper parameter tuning of some of the key parameters such as delay insensitivity, time step grouping and the active synapse grouping to achieve greater simulation speed of scalable spiking neural networks

Implementation and performance evaluation of a communication-avoiding GMRES method for stencil-based code on GPU cluster

In this study, a communication-avoiding generalized minimum residual method (CA-GMRES) is implemented on a hybrid CPU–GPU cluster, targeted for the performance acceleration of iterative linear system solver in the gyrokinetic toroidal five-dimensional Eulerian code (GT5D). In the GT5D, its sparse matrix-vector multiplication operation (SpMV) is performed as a 17-point stencil-based computation. The specialized part for the GT5D is only in the SpMV, and the other parts are usable also for other application program codes. In addition to the CA-GMRES, we implement and evaluate a modified variant of CA-GMRES (M-CA-GMRES) proposed in the previous study Idomura et al. (in: Proceedings of the 8th workshop on latest advances in scalable algorithms for large-scale systems (ScalA ’17), 2017. https://doi.org/10.1145/3148226.3148234) to reduce the amount of floating-point calculations. This study demonstrates that beneficial features of the CA-GMRES are in its minimum number of collective communications and its highly efficient calculations based on dense matrix–matrix operations. The performance evaluation is conducted on the Reedbush-L GPU cluster, which contains four NVIDIA Tesla P100 (Pascal GP100) GPUs per compute node. The evaluation results show that the M-CA-GMRES or CA-GMRES for the GT5D is advantageous over the GMRES or the generalized conjugate residual method (GCR) on GPU clusters, especially when the problem size (vector length) is large so that the cost of the SpMV is less dominant. The M-CA-GMRES is 1.09 ×, 1.22 × and 1.50 × faster than the CA-GMRES, GCR and GMRES, respectively, when 64 GPUs are used.

HyPar: A divide-and-conquer model for hybrid CPU–GPU graph processing

Efficient processing of graph applications on heterogeneous CPU–GPU systems require effectively harnessing the combined power of both the CPU and GPU devices. This paper presents HyPar, a divide-and-conquer model for processing graph applications on hybrid CPU–GPU systems. Our strategy partitions the given graph across the devices and performs simultaneous independent computations on both the devices. The model provides a simple and generic API, supported with efficient runtime strategies for hybrid executions. The divide-and-conquer model is demonstrated with five graph applications and using experiments with these applications on a heterogeneous system it is shown that our HyPar strategy provides equivalent performance to the state-of-art, optimized CPU-only and GPU-only implementations of the corresponding applications. When compared to the prevalent BSP approach for multi-device executions of graphs, our HyPar method yields 74%–92% average performance improvements.

Multiscale-optimized plasma turbulence simulation on petascale architectures

We describe the mathematical formulation, outline the numerical discretization, and present performance analysis results for the CGYRO plasma turbulence code. The performance data was collected on 5 current leadership systems (2 KNL-based, 2 hybrid CPU-GPU and 1 Skylake-based). CGYRO is a relatively new gyrokinetic turbulence code, based on the well-known GYRO code, but redesigned from the ground up to operate efficiently on multicore and GPU-accelerated systems. The gyrokinetic equations specify a 5-dimensional distribution function for each species, with species coupled through both the Maxwell equations and collision operator. For the cross-machine performance analysis, we report and compare timings for 8 separate computational kernels. This kernel-based breakdown illustrates the strengths and weaknesses of the floating-point and communication architectures of the respective systems. We conclude with a preview of new multiscale turbulence results that are shown to accurately recover experimentally-observed electron turbulence levels in an ITER-baseline plasma regime that cannot be described using traditional long-wavelength simulation.

Dual tree traversal on integrated GPUs for astrophysical N-body simulations

In astrophysical N-body simulations, O( N) fast multipole methods (FMMs) with dual tree traversal (DTT) on multi-core CPUs are faster than O( N log N) CPU tree-codes but can still be outperformed by GPU ones. In this article, we aim at combining the best algorithm, namely FMM with DTT, with the most powerful hardware currently available, namely GPUs. In the astrophysical context requiring low accuracies and non-uniform particle distributions, we show that such combination can be achieved thanks to a hybrid CPU-GPU algorithm on integrated GPUs: while the DTT is performed on the CPU cores, the far- and near-field computations are all performed on the GPU cores. We show how to efficiently expose the interactions resulting from the DTT to the GPU cores, how to deploy both the far- and near-field computations on GPU, and how to overlap the parallel DTT on CPU with GPU computations. Based on the falcON code and using OpenCL on AMD Accelerated Processing Units and on Intel integrated GPUs, this first heterogeneous deployment of DTT for FMM outperforms standard multi-core CPUs and matches GPU and high-end CPU performance, being hence more cost- and power-efficient.

Accelerating the task/data-parallel version of ILUPACK’s BiCG in multi-CPU/GPU configurations

ILUPACK is a valuable tool for the solution of sparse linear systems via iterative Krylov subspace-based methods. Its relevance for the solution of real problems has motivated several efforts to enhance its performance on parallel machines. In this work we focus on exploiting the task-level parallelism derived from the structure of the BiCG method, in addition to the data-level parallelism of the internal matrix computations, with the goal of boosting the performance of a GPU (graphics processing unit) implementation of this solver. First, we revisit the use of dual-GPU systems to execute independent stages of the BiCG concurrently on both accelerators, while leveraging the extra memory space to improve the data access patterns. In addition, we extend our ideas to compute the BiCG method efficiently in multicore platforms with a single GPU. In this line, we study the possibilities offered by hybrid CPU-GPU computations, as well as a novel synchronization-free sparse triangular linear solver. The experimental results with the new solvers show important acceleration factors with respect to the previous data-parallel CPU and GPU versions.

Novel Algorithm of CPU-GPU hybrid system for health care data classification

Due to advancements in portable health monitoring technology, such systems have become more and more economical & efficient. This in turn has resulted in a huge amount of data being generated every moment by millions of users of such portable devices. Such voluminous data may include audio, video, and image, and text representing blood pressure, temperature, vocal activity, ECG, sugar level etc. In the Proposed algorithm, first step is assignment, where clusters are assigned to a patient data and the second step is update, which takes the mean of the coordinates of all the data in its cluster. Medical practitioners and service providers can use such data to discover various patterns and useful insights. Such insights can be very useful on understanding various trends during epidemics, such as Malaria, Dengue, Chikungunya and other such outbreaks. A faster and economical way to get such insights is of paramount importance.
 Keywords: health monitoring; GPU; ECG; epidemics; data mining

A GPU based local search algorithm for the unweighted and weighted maximum s-plex problems

Given a graph G(V, E) and a value $$s \in {\mathbb {N}}$$, an s-plex S is a subset of V such that each vertex $$v \in S$$ has at least $$|S|-s$$ adjacent vertices in the subgraph induced by S. This work proposes a GPU based local search heuristic, called GPULS, for the problems of finding an s-plex of maximum cardinality and finding an s-plex of maximum weight. The proposed heuristic works well on both problems without any modification on its parameters or its code. GPULS considers two neighborhood structures, which are explored using tabu search and a first-improvement approach. We compare GPULS with the current best-performing exact methods and heuristics. The results obtained by GPULS are highly competitive, even when it runs on a CPU-only architecture. Moreover, we observed speedups of up to 16 times by running the heuristic on a hybrid CPU–GPU architecture.

DCA++: A software framework to solve correlated electron problems with modern quantum cluster methods

We present the first open release of the DCA++ project, a high-performance research software framework to solve quantum many-body problems with cutting edge quantum cluster algorithms. DCA++ implements the dynamical cluster approximation (DCA) and its DCA+ extension with a continuous self-energy. The algorithms capture nonlocal correlations in strongly correlated electron systems, thereby giving insight into high-Tc superconductivity. The code’s scalability allows efficient usage of systems at all scales, from workstations to leadership computers. With regard to the increasing heterogeneity of modern computing machines, DCA++ provides portable performance on conventional and emerging new architectures, such as hybrid CPU–GPU, sustaining multiple petaflops on ORNL’s Titan and CSCS’ Piz Daint supercomputers. Moreover, we show how sustainable and scalable development of the code base has been achieved by adopting standard techniques of the software industry. These include employing a distributed version control system, applying test-driven development and following continuous integration. Program summaryProgram Title: DCA++Program Files doi:http://dx.doi.org/10.17632/482jm5cv77.1Licensing provisions: BSD-3-ClauseProgramming language: C++14 and CUDANature of problem: Understanding the fascinating physics of strongly correlated electron systems requires the development of sophisticated algorithms and their implementation on leadership computing systems.Solution method: The DCA++ code provides a highly scalable and efficient implementation of the dynamical cluster approximation (DCA) and its DCA+ extension.