Parallel Overhead Research Articles

Research on the Application and Performance Optimization of GPU Parallel Computing in Concrete Temperature Control Simulation

With the development of engineering technology, engineering has higher requirements for the accuracy and the scale of simulation calculation. The computational efficiency of traditional serial programs cannot meet the requirements of engineering. Therefore, reducing the calculation time of the temperature control simulation program has important engineering significance for real-time simulation of temperature field and stress field, and then adopting more reasonable temperature control and crack prevention measures. GPU parallel computing is introduced into the temperature control simulation program of massive concrete to solve this problem and the optimization is carried out. Considering factors such as GPU clock rate, number of cores, parallel overhead and Parallel Region, the improved GPU parallel algorithm analysis indicator formula is proposed. It makes up for the shortcomings of traditional formulas that focus only on time. According to this formula, when there are enough threads, the parallel effect is limited by the size of the parallel domain, and when the parallel domain is large enough, the efficiency is limited by the parallel overhead and the clock rate. This paper studies the optimal Kernel execution configuration. Shared memory is utilized to improve memory access efficiency by 155%. After solving the problem of bank conflicts, an accelerate rate of 437.5× was realized in the subroutine of the matrix transpose of the solver. The asynchronous parallel of data access and logical operation is realized on GPU by using CUDA Stream, which can overlap part of the data access time. On the basis of GPU parallelism, asynchronous parallelism can double the computing efficiency. Compared with the serial program, the accelerate rate of inner product matrix multiplication of the GPU asynchronous parallel program is 61.42×. This study further proposed a theoretical formula of data access overlap rate to guide the selection of the number of CUDA streams to achieve the optimal computing conditions. The GPU parallel program compiled and optimized by the CUDA Fortran platform can effectively improve the computational efficiency of the simulation program for concrete temperature control, and better serve engineering computing.

Exploring temporal community evolution: algorithmic approaches and parallel optimization for dynamic community detection

Dynamic (temporal) graphs are a convenient mathematical abstraction for many practical complex systems including social contacts, business transactions, and computer communications. Community discovery is an extensively used graph analysis kernel with rich literature for static graphs. However, community discovery in a dynamic setting is challenging for two specific reasons. Firstly, the notion of temporal community lacks a widely accepted formalization, and only limited work exists on understanding how communities emerge over time. Secondly, the added temporal dimension along with the sheer size of modern graph data necessitates new scalable algorithms. In this paper, we investigate how communities evolve over time based on several graph metrics under a temporal formalization. We compare six different algorithmic approaches for dynamic community detection for their quality and runtime. We identify that a vertex-centric (local) optimization method works as efficiently as the classical modularity-based methods. To its advantage, such local computation allows for the efficient design of parallel algorithms without incurring a significant parallel overhead. Based on this insight, we design a shared-memory parallel algorithm DyComPar, which demonstrates between 4 and 18 fold speed-up on a multi-core machine with 20 threads, for several real-world and synthetic graphs from different domains.

Level-Based Blocking for Sparse Matrices: Sparse Matrix-Power-Vector Multiplication

The multiplication of a sparse matrix with a dense vector (SpMV) is a key component in many numerical schemes and its performance is known to be severely limited by main memory access. Several numerical schemes require the multiplication of a sparse matrix polynomial with a dense vector which is typically implemented as a sequence of SpMVs. This results in low performance and ignores the potential to increase the arithmetic intensity by reusing the matrix data from cache. In this work we use the recursive algebraic coloring engine (RACE) to enable blocking of sparse matrix data across the polynomial computations. In the graph representing the sparse matrix we form levels using a breadth-first search. Locality relations of these levels are then used to improve spatial and temporal locality when accessing the matrix data and to implement an efficient multithreaded parallelization. Our approach is independent of the matrix structure and avoids shortcomings of existing “blocking” strategies in terms of hardware efficiency and parallelization overhead. We quantify the quality of our implementation using performance modelling and demonstrate speedups of up to 3× and 5× compared to an optimal SpMV-based baseline on a single multicore chip of recent Intel and AMD architectures. Various numerical schemes like <inline-formula><tex-math notation="LaTeX">$s$</tex-math></inline-formula> -step Krylov solvers, polynomial preconditioners and power clustering algorithms will benefit from our development.

Accelerating Deep Learning Inference via Model Parallelism and Partial Computation Offloading

With the rapid development of Internet-of-Things (IoT) and the explosive advance of deep learning, there is an urgent need to enable deep learning inference on IoT devices in Mobile Edge Computing (MEC). To address the computation limitation of IoT devices in processing complex Deep Neural Networks (DNNs), computation offloading is proposed as a promising approach. Recently, partial computation offloading is developed to dynamically adjust task assignment strategy in different channel conditions for better performance. In this paper, we take advantage of intrinsic DNN computation characteristics and propose a novel Fused-Layer-based (FL-based) DNN model parallelism method to accelerate inference. The key idea is that a DNN layer can be converted to several smaller layers in order to increase partial computation offloading flexibility, and thus further create the better computation offloading solution. However, there is a trade-off between computation offloading flexibility as well as model parallelism overhead. Then, we investigate the optimal DNN model parallelism and the corresponding scheduling and offloading strategies in partial computation offloading. In particular, we propose a Particle Swarm Optimization with Minimizing Waiting (PSOMW) method, which explores and updates the FL strategy, path scheduling strategy, and path offloading strategy to reduce time complexity and avoid invalid solutions. Finally, we validate the effectiveness of the proposed method in commonly used DNNs. The results show that the proposed method can reduce the DNN inference time by an average of 12.75 times compared to the legacy No FL (NFL) algorithm, and is very close to the optimal solution achieved by the Brute Force (BF) algorithm with the difference of less than 0.04%.

SLF: A passive parallelization of subgraph isomorphism

Subgraph isomorphism is one of the most important graph query operations. Existing subgraph isomorphic parallelization methods suffer from redundant sharing or fine-grained parallelism. SLF, a parallel approach for subgraph isomorphism algorithm based on tree-search, is proposed in this paper. In SLF, threads are prevented from sharing tasks blindly by an “Ask for Sharing” mechanism. The “Low-depth Priority Sharing” rule filters out unnecessary sharing, which reduces the number of sharing. The context of shared tasks is represented by embedding, which reduces the time and memory usage of sharing. We prove that the parallelization overhead of SLF is smaller than that of previous method. The experimental results show that SLF achieves higher speedup and efficiency than state-of-the-art parallelization methods.

Reinforcement Learning-Based Delay-Aware Path Exploration of Parallelized Service Function Chains

The parallel processing of the service function chain (SFC) is expected to provide better low-delay service delivery, because it breaks through the bottleneck of traditional serial processing mode in which service delay increases linearly with the SFC length. However, the provision of parallelized SFC (PSFC) is much more difficult due to the unique construction of PSFCs, inevitable parallelization overhead, and delay balancing requirement of PSFC branches; therefore, existing mechanisms for serial SFC cannot be directly applied to PSFC. After a comprehensive review of recent related work, we find that traffic scheduling mechanisms for PSFCs is still lacking. In this paper, a delay-aware traffic scheduling mechanism (DASM) for PSFCs is proposed. DASM first transforms PSFC into several serial SFCs by releasing the upstream VNF constraints so as to handle them independently while keeping their parallel relations. Secondly, DASM realizes delay-aware PSFC traffic scheduling based on the reinforcement learning (RL) method. To the best knowledge of the authors, this is the first attempt to address the PSFC traffic scheduling problem by transforming them into independent serial SFCs. Simulation results show that the proposed DASM outperforms the advanced PSFCs scheduling strategies in terms of delay balance and throughput.

Scalable Graph Convolutional Network Training on Distributed-Memory Systems

Graph Convolutional Networks (GCNs) are extensively utilized for deep learning on graphs. The large data sizes of graphs and their vertex features make scalable training algorithms and distributed memory systems necessary. Since the convolution operation on graphs induces irregular memory access patterns, designing a memory- and communication-efficient parallel algorithm for GCN training poses unique challenges. We propose a highly parallel training algorithm that scales to large processor counts. In our solution, the large adjacency and vertex-feature matrices are partitioned among processors. We exploit the vertex-partitioning of the graph to use non-blocking point-to-point communication operations between processors for better scalability. To further minimize the parallelization overheads, we introduce a sparse matrix partitioning scheme based on a hypergraph partitioning model for full-batch training. We also propose a novel stochastic hypergraph model to encode the expected communication volume in mini-batch training. We show the merits of the hypergraph model, previously unexplored for GCN training, over the standard graph partitioning model which does not accurately encode the communication costs. Experiments performed on real-world graph datasets demonstrate that the proposed algorithms achieve considerable speedups over alternative solutions. The optimizations achieved on communication costs become even more pronounced at high scalability with many processors. The performance benefits are preserved in deeper GCNs having more layers as well as on billion-scale graphs.

High-performance and balanced parallel graph coloring on multicore platforms

Graph coloring is widely used to parallelize scientific applications by identifying subsets of independent tasks that can be executed simultaneously. Graph coloring assigns colors the vertices of a graph, such that no adjacent vertices have the same color. The number of colors used corresponds to the number of parallel steps in a real-world end-application. Therefore, the total runtime of the graph coloring kernel adds to the overall parallel overhead of the real-world end-application, whereas the number of the vertices of each color class determines the number of the independent concurrent tasks of each parallel step, thus affecting the amount of parallelism and hardware resource utilization in the execution of the real-world end-application. In this work, we propose a high-performance graph coloring algorithm, named ColorTM, that leverages Hardware Transactional Memory (HTM) to detect coloring inconsistencies between adjacent vertices. ColorTM detects and resolves coloring inconsistencies between adjacent vertices with an eager approach to minimize data access costs, and implements a speculative synchronization scheme to minimize synchronization costs and increase parallelism. We extend our proposed algorithmic design to propose a balanced graph coloring algorithm, named BalColorTM, with which all color classes include almost the same number of vertices to achieve high parallelism and resource utilization in the execution of the real-world end-applications. We evaluate ColorTM and BalColorTM using a wide variety of large real-world graphs with diverse characteristics. ColorTM and BalColorTM improve performance by 12.98times and 1.78times on average using 56 parallel threads compared to prior state-of-the-art approaches. Moreover, we study the impact of our proposed graph coloring algorithmic designs on a popular end-application, i.e., Community Detection, and demonstrate the ColorTM and BalColorTM can provide high-performance improvements in real-world end-applications across various input data given.

An efficient hardware supported and parallelization architecture for intelligent systems to overcome speculative overheads

In the last few decades, technology advancements have paved the way for the creation of intelligent and autonomous systems that utilize complex calculations which are both time-consuming and central processing unit intensive. As a consequence, parallel processing systems are gaining popularity to enhance overall computer performance. Programmers should be able to efficiently utilize available hardware resources with parallelization in an ideal world. Through the automatic parallelization of sequential code, multithreading can be executed without extra supervision. However, a wide range of software dependencies prevents this from being feasible. An architectural framework for speculative parallelization along with an efficient memory analysis and computational algorithms for the code generation are proposed that can provide optimal performance. Furthermore, a suitable support of hardware design as a runtime library to the proposed architectural framework is presented which can be used to recover misspeculated results during execution to minimize speculative parallelism overhead. The implementation makes use of the Low-Level Virtual Machine compiler infrastructure and is tested on numerous benchmarks, thus making it highly scalable in terms of programming languages and architectures. According to our experimental results, there is significant potential for speedup increase. In comparison to the overall function speedup, that is, geomean speedup of 5.2× approximately when using the proposed architecture without hardware support, the proposed architectural framework and algorithm with hardware support give an average geomean speedup of 7.0× approximately on the given benchmark which is written in C/C++.

Spatial parallel disturbance region update method with OpenMP for steady compressible flows

We present the spatial parallelization of the disturbance region update method (DRUM) with OpenMP, named the spatial parallel disturbance region update method (spDRUM). DRUM is an acceleration methodology for compressible flows, employing dynamic computational domains (DCDs) to eliminate the worthless computational effort of the conventional global-update method. However, the DCDs may cause a load-imbalance issue, making DRUM difficult to execute efficiently in parallel. In this work, parallel strategies of steps in spDRUM are introduced, and a dynamic load balancer that is established based on the equivalent number of cells in the DCDs and a combined grid partitioning method is proposed to exploit the full acceleration potential of spDRUM. Besides, a concept of sub-blocks that takes advantage of the memory-shared characteristic of OpenMP is embedded in the data structure to simplify the inner-boundary treatment. Numerical test cases demonstrate that spDRUM accomplishes remarkable convergence speed for solving all steady compressible flow problems; benefiting from the coupled contribution of parallelism and DCDs, the total speed-up ratio of spDRUM could be significantly greater than that of the ideal case ignoring parallel overheads and that of the global-update method with the same number of threads; the proposed grid partitioning method could generate various types of partitions efficiently and flexibly, keeping load balanced for multiblock structured grids; embedding the concept of sub-blocks in the data structure makes the number of inner-boundary cells unchanged and remains optimal, regardless of the variation in partitions.

Parallel domain discretization algorithm for RBF-FD and other meshless numerical methods for solving PDEs

In this paper, we present a novel parallel dimension-independent node positioning algorithm that is capable of generating nodes with variable density, suitable for meshless numerical analysis. A very efficient sequential algorithm based on Poisson disc sampling is parallelized for use on shared-memory computers, such as the modern workstations with multi-core processors. The parallel algorithm uses a global spatial indexing method with its data divided into two levels, which allows for an efficient multi-threaded implementation. The addition of bootstrapping enables the algorithm to use any number of parallel threads while remaining as general as its sequential variant. We demonstrate the algorithm performance on six complex 2- and 3-dimensional domains, which are either of non rectangular shape or have varying nodal spacing or both. We perform a run-time analysis of the algorithm, to demonstrate its ability to reach high speedups regardless of the domain and to show how well it scales on the experimental hardware with 16 processor cores. We also analyse the algorithm in terms of the effects of domain shape, quality of point placement, and various parallelization overheads.

Estimating the overhead and coupling of scientific computing clusters

Since simulation became the third pillar of scientific research, several forms of computers have become available to drive computer aided simulations, and nowadays, clusters are the most popular type of computers supporting these tasks. For instance, cluster settings, such as the so-called supercomputers, cluster of workstations (COW), cluster of desktops (COD), and cluster of virtual machines (COV) have been considered in literature to embrace a variety of scientific applications. However, those scientific applications categorized as high-performance computing (HPC) are conceptually restricted to be addressed only by supercomputers. In this aspect, we introduce the notions of cluster overhead and cluster coupling to assess the capacity of non-HPC systems to handle HPC applications. We also compare the cluster overhead with an existing measure of overhead in computing systems, the total parallel overhead, to explain the correctness of our methodology. The evaluation of capacity considers the seven dwarfs of scientific computing, which are well-known, scientific computing building blocks considered in the development of HPC applications. The evaluation of these building blocks provides insights regarding the strengths and weaknesses of non-HPC systems to deal with future HPC applications developed with one or a combination of these algorithmic building blocks.

Evaluation of automatic parallelization algorithms to minimize speculative parallelism overheads: An experiment

Automatic parallelization is a crucial objective of the parallel computing architecture that can be achieved through conversion of sequential code into multi-threaded code, which will run in parallel manner. This approach focuses largely on the loops since they take most of the execution time in programs. Thread level speculation techniques come into roleplay while checking for the dependencies. These dependencies cannot be identified at the compile time, thus providing a larger scope of parallelization when combined with other parallelisation techniques. This results in a greater speedup and more accurate parallel code formation. In this research paper, an experiment to evaluate the performance and comparative analysis has been done among key automatic parallelization algorithms on different parameters like number of cores, speedup and loop dependency taking into consideration of speculation.

Reduction of Magnetic Flux Density of Parallel Overhead Power Lines by Optimization of Phase Wires Hanging Order

Overhead power lines are one of the largest sources of the low-frequency magnetic fields. Such magnetic fields occur the harmful effects on human health and other biological objects. The current size of the security zones doesn’t provide a safe level of the magnetic flux density outside them. Therefore, the task of reducing the magnetic flux density value of the overhead power lines outside the security zone is urgent. Some overhead power lines have sections where two lines pass parallel to each other at a short distance. For such lines the method of phase wires hanging order optimization can be applied. For efficiency comparison of different hanging orders was chose two types of electricity pylons: portal (PB330-7) and triangular (P330-3). The cases both lines done by the same pylons were investigated. Analyze of different hanging orders efficiency was performed by mathematical simulation using MATLAB software. The magnetic flux density computation inside the security zone and outside it was done by a specialized simplified methodology of power lines electric and magnetic fields calculation. It is shown that in case of identical phase wires hanging order and the same currents direction of two parallel lines there is a significant increasing of magnetic flux density value at the border of the security zone and outside it for both types of the electricity pylons. It is determined that with a certain hanging order it is possible to achieve a significant reduction of the magnetic flux density value. For lines with electricity pylons of a portal type (PB330-7) the most efficient is the mirror order. It allows to obtain a decreasing of magnetic flux density more than 30 percent at the security zone border and more than 50 percent outside it. For pylons of a triangle type (P330-3) the most efficient is the nonsymmetrical order. It provides the decreasing of magnetic flux density up to 20 percent. In case of opposite currents direction, the most efficient is the same hanging order for both types of electricity pylons. It is shown that optimal phase wires hanging order allows to decrease the magnetic flux density value outside the security zone border, to reduce the distance to border of the area with safe level of the magnetic flux density, to increase the phase current value at which the magnetic flux density value reaches the safe level within the security zone.

Significance of Parallel Computing on the Performance of Digital Image Correlation Algorithms in MATLAB

Digital Image Correlation (DIC) is a powerful tool used to evaluate displacements and deformations in a non-intrusive manner. By comparing two images, one from the undeformed reference states of the sample and the other from the deformed target state, the relative displacement between the two states is determined. DIC is well-known and often used for post-processing analysis of in-plane displacements and deformation of the specimen. Increasing the analysis speed to enable real-time DIC analysis will be beneficial and expand the scope of this method. Here we tested several combinations of the most common DIC methods in combination with different parallelization approaches in MATLAB and evaluated their performance to determine whether the real-time analysis is possible with these methods. The effects of computing with different hardware settings were also analyzed and discussed. We found that implementation problems can reduce the efficiency of a theoretically superior algorithm, such that it becomes practically slower than a sub-optimal algorithm. The Newton–Raphson algorithm in combination with a modified particle swarm algorithm in parallel image computation was found to be most effective. This is contrary to theory, suggesting that the inverse-compositional Gauss–Newton algorithm is superior. As expected, the brute force search algorithm is the least efficient method. We also found that the correct choice of parallelization tasks is critical in attaining improvements in computing speed. A poorly chosen parallelization approach with high parallel overhead leads to inferior performance. Finally, irrespective of the computing mode, the correct choice of combinations of integer-pixel and sub-pixel search algorithms is critical for efficient analysis. The real-time analysis using DIC will be difficult on computers with standard computing capabilities, even if parallelization is implemented, so the suggested solution would be to use graphics processing unit (GPU) acceleration.

Accelerating Louvain community detection algorithm on graphic processing unit

The Louvain community detection algorithm is a hierarchal clustering method categorized in the NP-hard problem. Its execution time to find communities in large graphs is, therefore, a challenge. Parallelization is an effective solution for amortizing Louvain's execution time. In this paper, we propose an adaptive CUDA Louvain method (ACLM) algorithm that benefits from the graphic processing unit (GPU). ACLM uses the shared memory in GPU, as well as the optimal number of threads in the GPU blocks. These features minimize parallelization overhead and accelerate the calculation of modularity parameters. The proposed algorithm allocates threads to each block based on the number of required streaming multiprocessors (SMs) and warps on GPU. The implementation results show that ACLM can effectively accelerate the execution time by 77% compared to the competitive method in the large graph benchmarks.

A Recursive Algebraic Coloring Technique for Hardware-efficient Symmetric Sparse Matrix-vector Multiplication

The symmetric sparse matrix-vector multiplication (SymmSpMV) is an important building block for many numerical linear algebra kernel operations or graph traversal applications. Parallelizing SymmSpMV on today’s multicore platforms with up to 100 cores is difficult due to the need to manage conflicting updates on the result vector. Coloring approaches can be used to solve this problem without data duplication, but existing coloring algorithms do not take load balancing and deep memory hierarchies into account, hampering scalability and full-chip performance. In this work, we propose the recursive algebraic coloring engine (RACE), a novel coloring algorithm and open-source library implementation that eliminates the shortcomings of previous coloring methods in terms of hardware efficiency and parallelization overhead. We describe the level construction, distance- k coloring, and load balancing steps in RACE, use it to parallelize SymmSpMV, and compare its performance on 31 sparse matrices with other state-of-the-art coloring techniques and Intel MKL on two modern multicore processors. RACE outperforms all other approaches substantially. By means of a parameterized roofline model, we analyze the SymmSpMV performance in detail and discuss outliers. While we focus on SymmSpMV in this article, our algorithm and software are applicable to any sparse matrix operation with data dependencies that can be resolved by distance-k coloring.

Efficient space virtualization for the Hoshen–Kopelman algorithm

In this paper, the efficient space virtualisation for the Hoshen–Kopelman algorithm is presented. We observe minimal parallel overhead during computations, due to negligible communication costs. The proposed algorithm is applied for computation of random-site percolation thresholds for four dimensional simple cubic lattice with sites’ neighborhoods containing next–next–nearest neighbors (3NN). The obtained percolation thresholds are [Formula: see text], [Formula: see text], [Formula: see text], [Formula: see text], [Formula: see text], [Formula: see text], [Formula: see text], where 2NN and NN stand for next–nearest neighbors and nearest neighbors, respectively.

Study on Reducing the Parallel Overhead of the BBDF Method for Power System Transient Stability Simulations

In the power system parallel transient stability simulation (TSS), the parallel overhead will rise significantly with the increase in concurrency, which will seriously affect the efficiency of parallel computing. However, there are relatively few studies on the parallel overhead in the power system research field at present. In this paper, the geneses of the parallel overhead are introduced from the point of view of computer technology. The composition of the parallel overhead of the bordered block diagonal form (BBDF) method and the nested BBDF (NBBDF) method of the power system parallel TSS is analyzed. The time consumption of computing tasks and the parallel overhead of the BBDF method and the NBBDF method implemented by message passing interface (MPI) and OpenMP under different thread scheduling patterns are tested in detail in the 2383wp system and two practical power systems (13 490 node and 24 886 node). On this basis, an efficient implementation of the NBBDF method based on subnet-core mapping and mixed programming of MPI and OpenMP is suggested, which can further reduce the parallel overhead and increase the speedup ratio to more than 17x with 64 cores in the 24 886-node system. At last, some rules are summarized based on the test results.

Analysis and Optimization of Task Granularity on the Java Virtual Machine

Task granularity, i.e., the amount of work performed by parallel tasks, is a key performance attribute of parallel applications. On the one hand, fine-grained tasks (i.e., small tasks carrying out few computations) may introduce considerable parallelization overheads. On the other hand, coarse-grained tasks (i.e., large tasks performing substantial computations) may not fully utilize the available CPU cores, leading to missed parallelization opportunities. In this article, we provide a better understanding of task granularity for task-parallel applications running on a single Java Virtual Machine in a shared-memory multicore. We present a new methodology to accurately and efficiently collect the granularity of each executed task, implemented in a novel profiler (available open-source) that collects carefully selected metrics from the whole system stack with low overhead, and helps developers locate performance and scalability problems. We analyze task granularity in the DaCapo, ScalaBench, and Spark Perf benchmark suites, revealing inefficiencies related to fine-grained and coarse-grained tasks in several applications. We demonstrate that the collected task-granularity profiles are actionable by optimizing task granularity in several applications, achieving speedups up to a factor of 5.90×. Our results highlight the importance of analyzing and optimizing task granularity on the Java Virtual Machine.