CUDA Programming Model Research Articles

Slightly Off-Axis Digital Holography Using a Transmission Grating and GPU-Accelerated Parallel Phase Reconstruction

Slightly off-axis digital holography is proposed using transmission grating to obtain quantitative phase distribution. The experimental device is based on an improved 4f optical system in which a two-window input plane is used to form the object beam and reference beam. Then, the two beams are diffracted into multiple orders by the transmission grating placed at the Fourier plane. By applying a modified Michelson configuration, the interference patterns can be generated by the object and reference beams from different diffraction orders. After translating the grating, a random phase shift can be introduced to the hologram. To demonstrate the feasibility of our method, both thick and thin phase specimens are retrieved using two carrier phase-shifting holograms. Furthermore, we use the phase reconstruction algorithm based on the NVIDIA CUDA programming model to reduce the retrieval time. Meanwhile, we optimize the discrete cosine transform (DCT)-based least-squares unwrapping algorithm to unwrap the phase. By porting the entire phase reconstruction process to the graphics processing unit (GPU), the phase retrieval acceleration and execution efficiency significantly improve. To demonstrate the feasibility of our method, it is found that our method can measure the surface profiles of standard elements, such as a plano-convex cylinder lens and a microlens array, with a relative error of about 0.5%. For holograms with a different phase shift, the root-mean-square (RMS) value of the phase difference for the main imaging region is about 0.2 rad. By accelerating the phase reconstruction with GPU implementation, a speedup ratio of about 20× for the thick phase specimen and a speedup ratio of about 15× for the thin-phase specimen can be obtained for holograms with a pixel size of 1024 × 1024.

CuCatch: A Debugging Tool for Efficiently Catching Memory Safety Violations in CUDA Applications

CUDA, OpenCL, and OpenACC are the primary means of writing general-purpose software for NVIDIA GPUs, all of which are subject to the same well-documented memory safety vulnerabilities currently plaguing software written in C and C++. One can argue that the GPU execution environment makes software development more error prone. Unlike C and C++, CUDA features multiple, distinct memory spaces to map to the GPU’s unique memory hierarchy, and a typical CUDA program has thousands of concurrently executing threads. Furthermore, the CUDA platform has fewer guardrails than CPU platforms that have been forced to incrementally adjust to a barrage of security attacks. Unfortunately, the peculiarities of the GPU make it difficult to directly port memory safety solutions from the CPU space. This paper presents cuCatch, a new memory safety error detection tool designed specifically for the CUDA programming model. cuCatch combines optimized compiler instrumentation with driver support to implement a novel algorithm for catching spatial and temporal memory safety errors with low performance overheads. Our experimental results on a wide set of GPU applications show that cuCatch incurs a 19% runtime slowdown on average, which is orders of magnitude faster than state-of-the-art debugging tools on GPUs. Moreover, our quantitative evaluation demonstrates cuCatch’s higher error detection coverage compared to prior memory safety tools. The combination of high error detection coverage and low runtime overheads makes cuCatch an ideal candidate for accelerating memory safety debugging for GPU applications.

Multimobile Robot Cooperative Localization Using Ultrawideband Sensor and GPU Acceleration

To tackle the poor localization accuracy of multimobile robots caused by non-line-of-sight (NLOS) errors in a complex indoor environment and to meet the real-time requirement, this article proposes a multimobile robot cooperative localization system using ultrawideband (UWB) sensor and GPU hardware acceleration. First, a UWB multinode ranging network is established to obtain the relative distance information between robots and anchors. Then, the line-of-sight (LOS) and NLOS errors in distance information are effectively mitigated by using the proposed UWB ranging error mitigation algorithm based on the Bayesian filter. A cooperative particle filter (PF) localization algorithm based on the Gibbs sampling is designed to estimate the position information of each robot at any time. Finally, in order to improve the real-time performance of the collaborative localization system, a parallel Gibbs collaborative localization algorithm that can be accelerated by GPU is proposed considering the characteristics of GPU hardware and CUDA programming model. The experimental results of three TurtleBot2 mobile robots in real scene show that the proposed multimobile robot cooperative localization system using UWB technology can estimate the position information of each robot robustly and accurately, and the localization accuracy is superior to that of the popular extended Kalman filter (EKF) and PF algorithms. It is shown through further evaluations that the proposed parallel algorithm achieves about 3.2 times acceleration effect in the scenarios of three mobile robots. The speed gain is found more significant with more robots, which substantially improves the real-time performance of the cooperative localization system. In the test with seven mobile robots, the speedup is as high as 11.9, that is, the execution time of the algorithm is only 8.39% of that of the original algorithm. Note to Practitioners—The purpose of this article is to improve the accuracy and real-time indoor multimobile robot cooperative localization, but the method proposed in this article is also applicable to outdoor multimobile robot cooperative localization. The existing methods for indoor cooperative localization of mobile robots usually use Bluetooth, infrared, RFID, and other technologies to establish a wireless sensor network (WSN) and then combine Karman filter or particle filter (PF) to achieve cooperative localization, which is difficult to achieve low-cost and high-precision real-time localization. In this article, a new method of cooperative localization is proposed, which uses ultrawideband (UWB) ranging network with high penetration and high precision to obtain accurate distance information, then weakens NLOS error by the Bayesian filtering to further improve the accuracy of distance information, and, finally, uses a novel cooperative localization approach to realize fast and high-precision indoor multimobile robot localization. In addition, by redesigning the collaborative localization algorithm in parallel, the real-time performance of the algorithm is improved while ensuring high accuracy. The collaborative localization experiment of three mobile robots in the real scene shows that the proposed algorithm can effectively improve the localization accuracy, and the real-time performance of collaborative localization is significantly improved. However, the algorithm still has some limitations when mitigating UWB ranging errors in highly obstructed environments, and the algorithm parallelization framework can be further improved for higher real-time performance. In the future, we will further improve the robustness of cooperative localization and apply this algorithm to cooperative control of multiple mobile robots, such as formation control and cooperative search.

COX : Exposing CUDA Warp-level Functions to CPUs

As CUDA becomes the de facto programming language among data parallel applications such as high-performance computing or machine learning applications, running CUDA on other platforms becomes a compelling option. Although several efforts have attempted to support CUDA on devices other than NVIDIA GPUs, due to extra steps in the translation, the support is always a few years behind CUDA’s latest features. In particular, the new CUDA programming model exposes the warp concept in the programming language, which greatly changes the way the CUDA code should be mapped to CPU programs. In this article, hierarchical collapsing that correctly supports CUDA warp-level functions on CPUs is proposed. To verify hierarchical collapsing , we build a framework, COX , that supports executing CUDA source code on the CPU backend. With hierarchical collapsing , 90% of kernels in CUDA SDK samples can be executed on CPUs, much higher than previous works (68%). We also evaluate the performance with benchmarks for real applications and show that hierarchical collapsing can generate CPU programs with comparable or even higher performance than previous projects in general.

Towards Enhancing Coding Productivity for GPU Programming Using Static Graphs

The main contribution of this work is to increase the coding productivity of GPU programming by using the concept of Static Graphs. GPU capabilities have been increasing significantly in terms of performance and memory capacity. However, there are still some problems in terms of scalability and limitations to the amount of work that a GPU can perform at a time. To minimize the overhead associated with the launch of GPU kernels, as well as to maximize the use of GPU capacity, we have combined the new CUDA Graph API with the CUDA programming model (including CUDA math libraries) and the OpenACC programming model. We use as test cases two different, well-known and widely used problems in HPC and AI: the Conjugate Gradient method and the Particle Swarm Optimization. In the first test case (Conjugate Gradient) we focus on the integration of Static Graphs with CUDA. In this case, we are able to significantly outperform the NVIDIA reference code, reaching an acceleration of up to 11× thanks to a better implementation, which can benefit from the new CUDA Graph capabilities. In the second test case (Particle Swarm Optimization), we complement the OpenACC functionality with the use of CUDA Graph, achieving again accelerations of up to one order of magnitude, with average speedups ranging from 2× to 4×, and performance very close to a reference and optimized CUDA code. Our main target is to achieve a higher coding productivity model for GPU programming by using Static Graphs, which provides, in a very transparent way, a better exploitation of the GPU capacity. The combination of using Static Graphs with two of the current most important GPU programming models (CUDA and OpenACC) is able to reduce considerably the execution time w.r.t. the use of CUDA and OpenACC only, achieving accelerations of up to more than one order of magnitude. Finally, we propose an interface to incorporate the concept of Static Graphs into the OpenACC Specifications.

A Hybrid Multi-GPU Implementation of Simplex Algorithm with CPU Collaboration

The simplex algorithm has been successfully used for many years in solving linear programming (LP) problems. Due to the intensive computations required (especially for the solution of large LP problems), parallel approaches have also extensively been studied. The computational power provided by the modern GPUs as well as the rapid development of multicore CPU systems have led OpenMP and CUDA programming models to the top preferences during the last years. However, the desired efficient collaboration between CPU and GPU through the combined use of the above programming models is still considered a hard research problem. In the above context, we demonstrate here an excessively efficient implementation of standard simplex, targeting to the best possible exploitation of the concurrent use of all the computing resources, on a multicore platform with multiple CUDA-enabled GPUs. More concretely, we present a novel hybrid collaboration scheme which is based on the concurrent execution of suitably spread CPU-assigned (via multithreading) and GPU-offloaded computations. The experimental results extracted through the cooperative use of OpenMP and CUDA over a notably powerful modern hybrid platform (consisting of 32 cores and two high-spec GPUs – Titan Rtx and Rtx 2080Ti) highlight that the performance of the presented here hybrid GPU/CPU collaboration scheme is clearly superior to the GPU-only implementation under almost all conditions. The corresponding measurements validate the value of using all resources concurrently, even in the case of a multi-GPU configuration platform. Furthermore, the given implementations are completely comparable (and slightly superior in most cases) to other related attempts in the bibliography, and clearly superior to the native CPU implementation with 32 cores. Keywords—Parallel Computing; Linear Programming; Simplex Method; Multicore Platform; Hardware Acceleration, GPGPU; OpenMP; CUDA

Load balancing method for heterogeneous CFD algorithms

Abstract. The problem of load balancing for unstructured heterogeneous numerical algorithms for simulation of physical processes is considered. A computational distribution method for hybrid supercomputers with multicore CPUs and massively parallel accelerators is described. The load balancing procedure includes determination of dual graph vertices and edges weights, devices’ performance test and two-level decomposition of the computational mesh based on domain decomposition method. First level decomposition involves the graph partitioning between supercomputer nodes. On the second level node subdomains are partitioned between the MPI- processes running on the nodes. The details of the proposed approach are considered on the example of an unstructured finite-volume algorithm for modeling the Navier-Stokes equations with polynomial reconstruction of variables and explicit time integration scheme. The parallel version of the algorithm is developed using the MPI, OpenMP and CUDA programming models. The parameters of performance, parallel efficiency and scalability of the heterogeneous program are given. The results mentioned are obtained during the simulation of a supersonic flow around a sphere on a mixed mesh consisting of tetrahedrons, triangular prisms, quadrangular pyramids and hexagons.

GPU-based parallel Shadow Features generation at neural system for improving gait human activity recognition

In this paper, we propose a new method for improving human activity recognition (HAR) datasets in order to increase their classification accuracy when trained with a certain classifier like a Neural Network. In this paper a novel training/testing process for building/testing a classification model for human activity recognition (HAR) is proposed. Traditionally, HAR is done by a classifier that learns what activities a person is doing by training with skeletal data obtained from a motion sensor such as Microsoft Kinect or accelerometer sensors. These skeletal data are the spatial coordinates (x, y, z) of different parts of the human body. In addition to the spatial features that describe current positions in the skeletal data, new features called Shadow Features are used to improve the supervised learning efficiency and accuracy of Neural Network classifiers. Shadow Features are inferred from the dynamics of body movements, thereby modelling the underlying momentum of the performed activities. They provide extra dimensions of information for characterizing activities in the classification process and thus significantly improving the accuracy. These Shadow Features are generated based on the existing features obtained from sensor datasets. In this paper we show that the accuracy of a neural network classifier can be significantly improved by the addition of Shadow Features and we also show that the generation of Shadow Features can be achieved with little time cost, on the fly, with the NVIDIA GPU technology and the CUDA programming model, this way we can improve the Neural Network accuracy at almost no time cost. GPUs are particularly suitable for generating Shadow Features, since they possess multiple cores which can be taken advantage of, in order to generate Shadow Features for multiple data columns in parallel, therefore reducing a lot of processing time, especially when dealing with huge datasets.

Performance Analysis of Accelerator Architectures and Programming Models for Parareal Algorithm Solutions of Ordinary Differential Equations

Increasing needs for the study of complex dynamical systems require computing solutions of a large number of ordinary and partial differential time-dependent equations in near real-time. Numerical integration algorithms, which are computationally expensive and inherently sequential, are typically used to compute solutions of ordinary and partial differential time-dependent equations. This presents challenges to study complex dynamical systems in near real-time. This paper examines the challenges of computing solutions of ordinary differential time-dependent equations using the Parareal algorithm belonging to the class of parallel-in-time algorithms on various high-performance computing accelerator-based architectures and associated programming models. The paper presents the code refactoring steps and performance analysis of the Parareal algorithm on two accelerator computing architectures: the Intel Xeon Phi CPU and Graphics Processing Unit many-core architectures, and with OpenMP, OpenACC, and CUDA programming models. The speedup and scaling performance analysis are used to demonstrate the suitability of the Parareal to compute the solutions of a single ordinary differential time-dependent equation and a family of interdependent ordinary differential time-dependent. The speedup, weak and strong scaling results demonstrate the suitability of Graphical Processing Units with the CUDA programming model as the most efficient accelerator for computing solutions of ordinary differential time-dependent equations using parallel-in-time algorithms. Considering the time and effort required to refactor the code for execution on the accelerator architectures, the Graphical Processing Units with the OpenACC programming model is the most efficient accelerator for computing solutions of ordinary differential time-dependent equations using parallel-in-time algorithms.

Evaluation of Portable Acceleration Solutions for LArTPC Simulation Using Wire-Cell Toolkit

The Liquid Argon Time Projection Chamber (LArTPC) technology plays an essential role in many current and future neutrino experiments. Accurate and fast simulation is critical to developing efficient analysis algorithms and precise physics model projections. The speed of simulation becomes more important as Deep Learning algorithms are getting more widely used in LArTPC analysis and their training requires a large simulated dataset. Heterogeneous computing is an efficient way to delegate computationally intensive tasks to specialized hardware. However, as the landscape of compute accelerators quickly evolves, it becomes increasingly difficult to manually adapt the code to the latest hardware or software environments. A solution which is portable to multiple hardware architectures without substantially compromising performance would thus be very beneficial, especially for long-term projects such as the LArTPC simulations. In search of a portable, scalable and maintainable software solution for LArTPC simulations, we have started to explore high-level portable programming frameworks that support several hardware backends. In this paper, we present our experience porting the LArTPC simulation code in the Wire-Cell Toolkit to NVIDIA GPUs, first with the CUDA programming model and then with a portable library called Kokkos. Preliminary performance results on NVIDIA V100 GPUs and multi-core CPUs are presented, followed by a discussion of the factors affiecting the performance and plans for future improvements.

Performance analysis of CUDA, OpenACC and OpenMP programming models on TESLA V100 GPU

Graphics processors are widely utilized in modern supercomputers as accelerators. Ability to perform efficient parallelization and low-level allow scientists to greatly boost performance of their codes. Modern Nvidia GPUs feature low-level approaches, such as CUDA, along with high-level approaches: OpenACC and OpenMP. While the low-level approach aims to explore all possible abilities of SIMT GPU architecture by writing low-level C/C++ code, it takes significant effort from programmer. OpenACC and OpenMP programming models are opposite to CUDA. Using these models the programmer only have to identify the blocks of code to be parallelized using pragmas. We compare the performance of CUDA, OpenMP and OpenACC on state-of-the-art Nvidia Tesla V100 GPU in various typical scenarios that arise in scientific programming, such as matrix multiplication, regular memory access patterns and evaluate performance of physical simulation codes implemented using these programming models. Moreover, we study the performance matrix multiplication implemented in vendor-optimized BLAS libraries for Nvidia Tesla V100 GPU and modern Intel Xeon processor.

Optimal Kernel Design for Finite-Element Numerical Integration on GPUs

This article presents the design and optimization of the GPU kernels for numerical integration, as it is applied in the standard form in finite-element codes. The optimization process employs autotuning, with the main emphasis on the placement of variables in the shared memory or registers. OpenCL and the first order finite-element method (FEM) approximation are selected for code design, but the techniques are also applicable to the CUDA programming model and other types of finite-element discretizations (including discontinuous Galerkin and isogeometric). The autotuning optimization is performed for four example graphics processors and the obtained results are discussed.

Multi-GPU Island-Based Genetic Algorithm

Genetic algorithms are effective in solving many optimization tasks. However, the long execution time associated with it prevents its use in many domains. In this paper, we propose a new approach for parallel implementation of genetic algorithm on graphics processing units (GPUs) using CUDA programming model. This paper introduces a novel implementation of the genetic algorithm exploiting a multi-GPU cluster. The proposed implementation employs an island-based genetic algorithm where every GPU evolves a single island. Recently there has been a growing interest in developing parallel algorithms using graphic processing units (GPU) also referred as GPU computation. Advances in the video gaming industry have led to the production of low-cost, high performance graphics processing units that possess more memory bandwidth and computational capability than central processing units (CPUs). As GPUs are available in personal computers, and they are easy to use and manage through several GPU programming languages, graphics engines are being adopted widely in scientific computing.

Accelerated iterative image reconstruction for cone-beam computed tomography through Big Data frameworks

One of the latest trends in Computed Tomography (CT) is the reduction of the radiation dose delivered to patients through the decrease of the amount of acquired data. This reduction results in artifacts in the final images if conventional reconstruction methods are used, making it advisable to employ iterative algorithms to enhance image quality. Most approaches are built around two main operators, backprojection and projection, which are computationally expensive.In this work, we present an implementation of those operators for iterative reconstruction methods exploiting the Big Data paradigm. We define an architecture based on Apache Spark that supports both Graphical Processing Units (GPU) and CPU-based architectures. The aforementioned are parallelized using a partitioning scheme based on the division of the volume and irregular data structures in order to reduce the cost of communication and computation of the final images. Our solution accelerates the execution of the two most computational expensive components with Apache Spark, improving the programming experience of new iterative reconstruction algorithms and the maintainability of the source code increasing the level of abstraction for non-experienced high performance programmers. Through an experimental evaluation, we show that we can obtain results up to 10× faster for projection and 21× faster for backprojection when using a GPU-based cluster compared to a traditional multi-core version. Although a linear speed up was not reached, the proposed approach can be a good alternative for porting previous medical image reconstruction applications already implemented in C/C++ or even with CUDA or OpenCL programming models. Our solution enables the automatic detection of the GPU devices and execution on CPU and GPU tasks at the same time under the same system, using all the available resources.

On GPU–CUDA as preprocessing of fuzzy-rough data reduction by means of singular value decomposition

Data reduction algorithms often produce inaccurate results for loss of relevant information. Recently, the singular value decomposition (SVD) method has been used as preprocessing method in order to deal with high-dimensional data and achieve fuzzy-rough reduct convergence on higher dimensional datasets. Despite the well-known fact that SVD offers attractive properties, its high computational cost remains a critical issue. In this work, we present a parallel implementation of the SVD algorithm on graphics processing units using CUDA programming model. Our approach is based on an iterative parallel version of the QR factorization by means of Givens rotations using the Sameh and Kuck scheme. Our results show significant improvements in terms of performances with respect to the CPU version that encourage its usability for this expensive processing of data.

FUX-Sim: Implementation of a fast universal simulation/reconstruction framework for X-ray systems.

The availability of digital X-ray detectors, together with advances in reconstruction algorithms, creates an opportunity for bringing 3D capabilities to conventional radiology systems. The downside is that reconstruction algorithms for non-standard acquisition protocols are generally based on iterative approaches that involve a high computational burden.The development of new flexible X-ray systems could benefit from computer simulations, which may enable performance to be checked before expensive real systems are implemented. The development of simulation/reconstruction algorithms in this context poses three main difficulties. First, the algorithms deal with large data volumes and are computationally expensive, thus leading to the need for hardware and software optimizations. Second, these optimizations are limited by the high flexibility required to explore new scanning geometries, including fully configurable positioning of source and detector elements. And third, the evolution of the various hardware setups increases the effort required for maintaining and adapting the implementations to current and future programming models. Previous works lack support for completely flexible geometries and/or compatibility with multiple programming models and platforms.In this paper, we present FUX-Sim, a novel X-ray simulation/reconstruction framework that was designed to be flexible and fast. Optimized implementation for different families of GPUs (CUDA and OpenCL) and multi-core CPUs was achieved thanks to a modularized approach based on a layered architecture and parallel implementation of the algorithms for both architectures.A detailed performance evaluation demonstrates that for different system configurations and hardware platforms, FUX-Sim maximizes performance with the CUDA programming model (5 times faster than other state-of-the-art implementations). Furthermore, the CPU and OpenCL programming models allow FUX-Sim to be executed over a wide range of hardware platforms.

GPU accelerated implementation of NCI calculations using promolecular density.

The NCI approach is a modern tool to reveal chemical noncovalent interactions. It is particularly attractive to describe ligand-protein binding. A custom implementation for NCI using promolecular density is presented. It is designed to leverage the computational power of NVIDIA graphics processing unit (GPU) accelerators through the CUDA programming model. The code performances of three versions are examined on a test set of 144 systems. NCI calculations are particularly well suited to the GPU architecture, which reduces drastically the computational time. On a single compute node, the dual-GPU version leads to a 39-fold improvement for the biggest instance compared to the optimal OpenMP parallel run (C code, icc compiler) with 16 CPU cores. Energy consumption measurements carried out on both CPU and GPU NCI tests show that the GPU approach provides substantial energy savings. © 2017 Wiley Periodicals, Inc.

GPU-accelerated iterative solution of complex-entry systems issued from 3D edge-FEA of electromagnetics in the frequency domain

We present a performance analysis of a parallel implementation for both preconditioned conjugate gradient and preconditioned bi-conjugate gradient solvers running on graphic processing units (GPUs) with CUDA programming model. The solvers were mainly optimized for the solution of sparse systems of algebraic equations at complex entries, arising from the three-dimensional edge-finite element analysis of the electromagnetic phenomena involved in the open-bound earth diffusion of currents under time-harmonic excitation. We used a shifted incomplete Cholesky (IC) factorization as preconditioner. Results show a significant speedup by using either a single-GPU or a multi-GPU device, compared to a serial central processing unit (CPU) implementation, thereby allowing the simulations of large-scale problems in low-cost personal computers. Additional experiments of the optimized solvers show that its use can be extended successfully to other complex systems of equations arising in electrical engineering, such as those obtained in power–system analysis.

Discovery of semantic associations in an RDF graph using bi-directional BFS on massively parallel hardware

Resource description framework (RDF) data model provides a framework to capture the meaning of an entity by specifying how it relates to other entities. Large RDF graph involving millions of entities are common in many semantic graph applications and are challenging to process. For example, finding complex relationships called semantic associations between two entities in an RDF graph is a tedious process. Graphics processing units (GPUs) provides high computation power at low price. Today, the GPUs expose a general data-parallel programming model in the form of CUDA. In this paper, we present the implementation of bi-directional breadth-first-search algorithm to discover the semantic association in RDF graph using CUDA programming model. In the experimental evaluation, we prove that our proposed algorithm is faster than the existing algorithms.

Discrete element simulation of mill charge in 3D using the BLAZE-DEM GPU framework

The Discrete Element Method (DEM) simulation of charge motion in ball, semi-autogenous (SAG) and autogenous mills has advanced to a stage where the effects of lifter design, power draft and product size can be evaluated with sufficient accuracy using either two-dimensional (2D) or three-dimensional (3D) codes. While 2D codes may provide a reasonable profile of charge distribution in the mill there is a difference in power estimations as the anisotropic nature within the mill cannot be neglected. Thus 3D codes are preferred as they can provide a more accurate estimation of power draw and charge distribution. While 2D codes complete a typical industrial simulation in the order of hours, 3D codes require computing times in the order of days to weeks on a typical multi-threaded desktop computer. A newly developed and recently introduced 3D DEM simulation environment is BLAZE-DEM that utilizes the Graphical Processor Unit (GPU) via the NVIDIA CUDA programming model. Utilizing the parallelism of the GPU a 3D simulation of an industrial mill with four million particles takes 1h to simulate one second (20 FPS) on a GTX 880 laptop GPU. This new performance level may allow 3D simulations to become a routine task for mill designers and researchers. This paper makes two notable extensions to the BLAZE-DEM environment. Firstly, the sphere-face contact is extended to include a GPU efficient sphere-edge contact strategy. Secondly, the world representation is extended by an efficient representation of convex geometrical primitives that can be combined to form non-convex world boundaries that drastically enhances the efficiency of particle world contact. In addition to these extensions this paper verifies and validates our GPU code by comparing charge profiles and power draw obtained using the CPU based code Millsoft and pilot scale experiments. Finally, we conclude with plant scale mill simulations.