Fine-grained Parallelism Research Articles

Construction of a Mental Health Education Model for College Students Based on Fine-Grained Parallel Computing Programming

Mental health and mental health problems of college students are becoming more and more obvious, and there is more and more negative news caused by psychological problems, and society from all walks of life has given high attention to this problem. Given the new situations and new problems, how to keep up with the times and reform and innovate in the content, method, and path of psychological education in colleges and universities is an important work of ideological and political education in colleges and universities. Because fine-grained category information can provide rich semantic clues, fine-grained parallel computing techniques are widely used in tasks such as sensitive feature filtering, medical image classification, and dangerous goods detection. In this study, we adopt a fine-grained parallel computing programming approach and propose a multiobjective matrix regular optimization algorithm that can simultaneously perform the joint square root, low-rank, and sparse regular optimization for bilinear visual features, which is used to stabilize the higher-order semantic information in bilinear features, improve the generalization ability of features, and apply it to the construction of mental health education models for college students to promote the construction of mental health education bases, improve mental health education network platform, and strengthen the construction of mental health education data platform. A new practical aspect has been added to the abstract. The saliency-guided data augmentation method in this study is an improvement on random data augmentation but reduces the randomness in the data augmentation process and significantly improves the results. The best result belongs to SCutMix data augmentation, which improves by 1.9% compared to the baseline network.

FgSpMSpV: A Fine-grained Parallel SpMSpV Framework on HPC Platforms

Sparse matrix-sparse vector (SpMSpV) multiplication is one of the fundamental and important operations in many high-performance scientific and engineering applications. The inherent irregularity and poor data locality lead to two main challenges to scaling SpMSpV over high-performance computing (HPC) systems: (i) a large amount of redundant data limits the utilization of bandwidth and parallel resources; (ii) the irregular access pattern limits the exploitation of computing resources. This paper proposes a fine-grained parallel SpMSpV (fgSpMSpV) framework on Sunway TaihuLight supercomputer to alleviate the challenges for large-scale real-world applications. First,fgSpMSpVadopts an MPI\( + \)OpenMP\( +X \)parallelization model to exploit the multi-stage and hybrid parallelism of heterogeneous HPC architectures and accelerate both pre-/post-processing and main SpMSpV computation. Second,fgSpMSpVutilizes an adaptive parallel execution to reduce the pre-processing, adapt to the parallelism and memory hierarchy of the Sunway system, while still tame redundant and random memory accesses in SpMSpV, including a set of techniques like the fine-grained partitioner, re-collection method, and Compressed Sparse Column Vector (CSCV) matrix format. Third,fgSpMSpVuses several optimization techniques to further utilize the computing resources.fgSpMSpVon the Sunway TaihuLight gains a noticeable performance improvement from the key optimization techniques with various sparsity of the input. Additionally,fgSpMSpVis implemented on an NVIDIA Tesal P100 GPU and applied to the breath-first-search (BFS) application.fgSpMSpVon a P100 GPU obtains the speedup of up to\( 134.38\times \)over the state-of-the-art SpMSpV algorithms, and the BFS application usingfgSpMSpVachieves the speedup of up to\( 21.68\times \)over the state-of-the-arts.

Enhanced Parallel Sine Cosine Algorithm for Constrained and Unconstrained Optimization

The sine cosine algorithm’s main idea is the sine and cosine-based vacillation outwards or towards the best solution. The first main contribution of this paper proposes an enhanced version of the SCA algorithm called as ESCA algorithm. The supremacy of the proposed algorithm over a set of state-of-the-art algorithms in terms of solution accuracy and convergence speed will be demonstrated by experimental tests. When these algorithms are transferred to the business sector, they must meet time requirements dependent on the industrial process. If these temporal requirements are not met, an efficient solution is to speed them up by designing parallel algorithms. The second major contribution of this work is the design of several parallel algorithms for efficiently exploiting current multicore processor architectures. First, one-level synchronous and asynchronous parallel ESCA algorithms are designed. They have two favors; retain the proposed algorithm’s behavior and provide excellent parallel performance by combining coarse-grained parallelism with fine-grained parallelism. Moreover, the parallel scalability of the proposed algorithms is further improved by employing a two-level parallel strategy. Indeed, the experimental results suggest that the one-level parallel ESCA algorithms reduce the computing time, on average, by 87.4% and 90.8%, respectively, using 12 physical processing cores. The two-level parallel algorithms provide extra reductions of the computing time by 91.4%, 93.1%, and 94.5% with 16, 20, and 24 processing cores, including physical and logical cores. Comparison analysis is carried out on 30 unconstrained benchmark functions and three challenging engineering design problems. The experimental outcomes show that the proposed ESCA algorithm behaves outstandingly well in terms of exploration and exploitation behaviors, local optima avoidance, and convergence speed toward the optimum. The overall performance of the proposed algorithm is statistically validated using three non-parametric statistical tests, namely Friedman, Friedman aligned, and Quade tests.

GSoFa: Scalable Sparse Symbolic LU Factorization on GPUs

Decomposing a matrix <inline-formula><tex-math notation="LaTeX">$\mathbf {A}$</tex-math></inline-formula> into a lower matrix <inline-formula><tex-math notation="LaTeX">$\mathbf {L}$</tex-math></inline-formula> and an upper matrix <inline-formula><tex-math notation="LaTeX">$\mathbf {U}$</tex-math></inline-formula> , which is also known as LU decomposition, is an essential operation in numerical linear algebra. For a sparse matrix, LU decomposition often introduces more nonzero entries in the <inline-formula><tex-math notation="LaTeX">$\mathbf {L}$</tex-math></inline-formula> and <inline-formula><tex-math notation="LaTeX">$\mathbf {U}$</tex-math></inline-formula> factors than in the original matrix. A <i>symbolic factorization</i> step is needed to identify the nonzero structures of <inline-formula><tex-math notation="LaTeX">$\mathbf {L}$</tex-math></inline-formula> and <inline-formula><tex-math notation="LaTeX">$\mathbf {U}$</tex-math></inline-formula> matrices. Attracted by the enormous potentials of the Graphics Processing Units (GPUs), an array of efforts have surged to deploy various LU factorization steps except for the symbolic factorization, to the best of our knowledge, on GPUs. This article introduces <small>gSoFa</small> , the first <u>G</u> PU-based <u>s</u> ymb <u>o</u> lic <u>fa</u> ctorization design with the following three optimizations to enable scalable LU symbolic factorization for <i>nonsymmetric pattern</i> sparse matrices on GPUs. First, we introduce a novel fine-grained parallel symbolic factorization algorithm that is well suited for the <i>Single Instruction Multiple Thread</i> (SIMT) architecture of GPUs. Second, we tailor supernode detection into a SIMT friendly process and strive to balance the workload, minimize the communication and saturate the GPU computing resources during supernode detection. Third, we introduce a three-pronged optimization to reduce the excessive space consumption problem faced by multi-source concurrent symbolic factorization. Taken together, <small>gSoFa</small> achieves up to 31× speedup from 1 to 44 Summit nodes (6 to 264 GPUs) and outperforms the state-of-the-art CPU project, on average, by 5×. Notably, <small>gSoFa</small> also achieves up to 47 percent of the peak memory throughput of a V100 GPU in the Summit Supercomputer.

GPU-accelerated N-1 static security analysis based on fine-grained parallelism HELM

With the increment of scale of power system, the accuracy and efficiency requirements of N-1 static security analysis (SSA) keep increasing. In order to satisfy these requirements, this paper proposed a GPU-accelerated N-1 SSA method to reduce computation burden and ensure the accuracy of alternating current power flow (ACPF) and connectivity test simultaneously. First, in the algorithm, ACPF is performed by holomorphic embedding load flow method (HELM). The construction and solution of sparse linear system both adopts fine-grained parallelism. According to the mathematical lemma of combining linear equations, this paper performs fine-grained parallelism Padé approximation of HELM. Second, considering the high time complexity of the original search-based algorithm and adjacency matrix method which is not suitable for sparse graph, this paper proposes a GPU-accelerated Union Set algorithm. Its main advantage is lower time complexity compared to other graph theory methods. And most importantly, it is very suitable for parallelization. Shared memory can be used in this algorithm well. Therefore, it is not complicated to perform a block-level parallelism in GPU-accelerated Union Set. Finally, the overall framework of GPU-accelerated N-1 SSA method is shown in this paper. It has been tested on large-scale power systems of up to 2869 buses. The computation results are compared with MATPOWER and the execution times are compared with other mainstream method to show the improvement in performance.

An Ultra-Fast Authenticated Encryption Scheme with Associated Data Using AES-OTR

Applications relying on the Ethernet IEEE 802.3ba and IEEE802.3b standards require secure data encryption and authentication at extremely high speeds (at least 100 Gbit/s). The Galois/Counter Mode (GCM) is currently considered the de facto standard for hardware high-speed authenticated encryption, although other algorithms have been proposed in the literature such as the Offset Codebook Mode (OCB). The challenge in terms of providing security for high-speed applications is to achieve implementations that explore the parallelism of these algorithms; however, this translates into area cost. In this work, we propose an alternative to GCM and OCB. We show that a combination of the Offset Two-Round authenticated-encryption scheme with the AES block cipher (known as AES-OTR) is exceptionally well suited for exploiting fine-grained parallelism, and can therefore be used to achieve ultra-high-speed data encryption rates. The experiments reported in this paper show that our pipeline-parallel implementation of AES-OTR outperforms the GCM and OCB schemes in terms of throughput per area while using almost half of the logic resources. Our implementation used a Stratix 4 FPGA device as well as several devices from the Virtex family. Implementations of AES-OTR on Stratix 4 used 11kALMs and achieved a throughput of 143.65 Gbit/s. On the Virtex Ultrascale, our design used 31,859 LUTs with a throughput of 204.92 Gbit/s.

Fine-grained parallelism framework with predictable work-stealing for real-time multiprocessor systems

Lately, parallel task models have received much attention in the development of real-time multiprocessor systems, as they allow highly compute-intensive tasks to have shorter deadlines which is very much required in modern reactive systems. However, missing modularity and portability can make parallel programming a cumbersome endeavor. As a consequence, compute-intensive sectors in the desktop and server segment have relied on parallelism frameworks such as Intel Threading Building Blocks, Cilk and OpenMP. These parallelism frameworks, however, are optimized for decent average case performance and consequently, do not meet the strict requirements imposed by real-time systems.In this paper, we present a proof-of-concept parallelism framework which was implemented in particular for soft real-time systems and having tight timing and safety requirements of such critical systems in mind. The proposed runtime system implements static memory allocation in a work-stealing environment that conforms to the strict space and tight probabilistic time bounds of work-stealing schedulers. Furthermore, we evaluate the performance of this framework by conducting multiprogrammed benchmarks on a real-time embedded multicore architecture.

Parallel electromagnetic transient simulation of power systems with a high proportion of renewable energy based on latency insertion method

With the interconnection of regional grids and the increasing penetration of renewable energy, the order of the power system model is getting higher and higher, which brings great challenges to the accuracy and real-time performance of electromagnetic transient (EMT) simulation. The simulation of power systems with a high proportion of renewable energy requires a sharp increase in the performance of the hardware platform and the parallelism of the simulation algorithm. To solve this problem, the paper proposes a fine-grained parallel modelling method for power systems with a high proportion of renewable energy, which is inspired by the latency insertion method used in the design of large-scale integrated circuits. And this method is implemented on GPU to achieve the component-level and device-level parallelized simulation of power systems with a high proportion of renewable energy. Finally, the correctness of the proposed algorithm and the CPU-GPU cooperative architecture is verified based on an improved 118-bus case. The cascade system of the improved 9-bus case is used as an example to verify the superiority of the proposed algorithm in terms of simulation efficiency.

Exploring the Approaches to Data Flow Computing

Architectures based on the data flow computing model provide an alternative to the conventional Von-Neumann architecture that are widely used for general purpose computing. Processors based on the data flow architecture employ fine-grain data-driven parallelism. These architectures have the potential to exploit the inherent parallelism in compute intensive applications like signal processing, image and video processing and so on and can thus achieve faster throughputs and higher power efficiency. In this paper, several data flow computing architectures are explored, and their main architectural features are studied. Furthermore, a classification of the processors is presented based on whether they employ either the data flow execution model exclusively or in combination with the control flow model and are accordingly grouped as exclusive data flow or hybrid architectures. The hybrid category is further subdivided as conjoint or accelerator-style architectures depending on how they deploy and separate the data flow and control flow execution model within their execution blocks. Lastly, a brief comparison and discussion of their advantages and drawbacks is also considered. From this study we conclude that although the data flow architectures are seen to have matured significantly, issues like data-structure handling and lack of efficient placement and scheduling algorithms have prevented these from becoming commercially viable.

On-Orbit Real-Time Variational Image Destriping: FPGA Architecture and Implementation

On-orbit real-time image processing is of increasing demands due to requirements for quick-response missions. Image destriping is usually an important pre-processing step to improve image quality in practice. The unidirectional variational models have shown impressive destriping performance. However, they are not easy for real-time implementation for high-computation complexity and there are few attempts for hardware implementation. This article is the first hardware implementation of variational image destriping algorithm, which achieves high throughput for large-swath remote-sensing images. In this article, a fully pipelined hardware architecture is proposed. First, the involved iteration loop is unrolled and a coarse-grained parallelism is obtained. Second, for each deployed iteration computation blocks (ICBs), a dedicated timing arrangement is designed to alleviate the bottleneck caused by the data dependency within each ICB, obtaining a fine-grained parallelism. Moreover, to further optimize the critical path, an approximate simplification scheme is proposed, saving the resource usage and reducing computing delay. The proposed architecture is implemented and verified on a XILINX 6vcx240t field programmable gate array (FPGA); it achieves a maximum frame rate up to 41.9 frames/s with delay of only tens of row cycles for 8-bit <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${{2048}} \times {{2048}}$ </tex-math></inline-formula> images. It performs all the processing on the pixels in raster scan order on- the-fly as they are being transmitted from camera payload, which significantly facilitates on- orbit real-time processing of large-swath remote-sensing images with high data rate.

Locality Sensitive Hash Aggregated Nonlinear Neighborhood Matrix Factorization for Online Sparse Big Data Analysis

Matrix factorization (MF) can extract the low-rank features and integrate the information of the data manifold distribution from high-dimensional data, which can consider the nonlinear neighborhood information. Thus, MF has drawn wide attention for low-rank analysis of sparse big data, e.g., Collaborative Filtering (CF) Recommender Systems, Social Networks, and Quality of Service. However, the following two problems exist: (1) huge computational overhead for the construction of the Graph Similarity Matrix (GSM) and (2) huge memory overhead for the intermediate GSM. Therefore, GSM-based MF, e.g., kernel MF, graph regularized MF, and so on, cannot be directly applied to the low-rank analysis of sparse big data on cloud and edge platforms. To solve this intractable problem for sparse big data analysis, we propose Locality Sensitive Hashing (LSH) aggregated MF (LSH-MF), which can solve the following problems: (1) The proposed probabilistic projection strategy of LSH-MF can avoid the construction of the GSM. Furthermore, LSH-MF can satisfy the requirement for the accurate projection of sparse big data. (2) To run LSH-MF for fine-grained parallelization and online learning on GPUs, we also propose CULSH-MF, which works on CUDA parallelization. Experimental results show that CULSH-MF can not only reduce the computational time and memory overhead but also obtain higher accuracy. Compared with deep learning models, CULSH-MF can not only save training time but also achieve the same accuracy performance.

Porting hypre to heterogeneous computer architectures: Strategies and experiences

Linear systems are occurring in many applications, and solving them can take a large amount of the total simulation time. The high performance library hypre provides a variety of interfaces and linear solvers, including various multigrid methods, that have achieved good scalability on a variety of homogeneous parallel computer architectures. Heterogeneous architectures with nodes that have both CPUs and accelerators provide new challenges, since they require more fine-grained parallelism and reduced data movement between different memories on a single node as well as across nodes. We will discuss our experiences and strategies to port hypre to heterogeneous computers with accelerators, including the design of a new memory model, the use of abstractions, the BoxLoop macros in the structured and semi-structured interfaces, and the restructuring of algebraic multigrid (AMG) into modular components. We present numerical experiments comparing CPU and GPU performance for several test problems.

Study of Fine-grained Nested Parallelism in CDCL SAT Solvers

Boolean satisfiability (SAT) is an important performance-hungry problem with applications in many problem domains. However, most work on parallelizing SAT solvers has focused on coarse-grained, mostly embarrassing, parallelism. Here, we study fine-grained parallelism that can speed up existing sequential SAT solvers, which all happen to be of the so-called Conflict-Directed Clause Learning variety. We show the potential for speedups of up to 382× across a variety of problem instances. We hope that these results will stimulate future research, particularly with respect to a computer architecture open problem we present.

Immortal rays: Rethinking random ray neutron transport on GPU architectures

The Random Ray Method (TRRM) is a recently developed adaptation of the Method of Characteristics for neutral particle transport simulations. TRRM has demonstrated excellent performance on 3D nuclear reactor benchmark problems using CPU-based compute systems. When porting to GPU-based systems, however, new performance challenges arise that are unique to processors targeting massive fine-grained parallelism. For smaller problems, or for large problems that are domain decomposed across many computational nodes, the problem size per node has insufficient parallelism to saturate GPU node resources, thus greatly limiting speedup. In this study, we report on a newly developed “immortal ray” variant of TRRM. The immortal ray technique exposes significantly more fine-grained parallelism by fundamentally reformulating the numerical details of ray discretization, resulting in performance tradeoffs with significant overall benefit on GPUs. For very small 2D simulation problems we found the new immortal ray variant allowed for up to a 4.4x speedup when run on a single GPU. For larger 3D simulation problems we found the new variant improved strong scaling by 3x when run on the Summit supercomputer.

A load balancing algorithm for the parallel automated multilevel substructuring method

The objective of this paper is to present a load balancing algorithm for the parallel automated multilevel substructuring (PAMLS) method. In the PAMLS method, load balancing is highly dependent on the computation time for the transformation and back transformation procedures corresponding to substructures. To balance the workload among threads, the proposed algorithm consists of two types of granularity: coarse-grained and fine-grained parallel algorithms. According to the level of substructures, the coarse-grained parallel algorithm splits both the transformation and back transformation procedures and assigns them to threads. Through fine-grained parallelism, more threads are exploited for the transformation of each substructure compared to threads used in the original PAMLS method. Without repartitioning, the proposed algorithm significantly improves the efficiency of the PAMLS method.

An Energy Efficient Accelerator for Bidirectional Recurrent Neural Networks (BiRNNs) Using Hybrid-Iterative Compression With Error Sensitivity

Recurrent Neural Networks (RNNs) have been widely used in many sequential applications, such as machine translation, speech recognition and sentiment analysis. Long Term Short Term Memory (LSTM) and Gated Recurrent Unit (GRU) are widely used variants of RNN due to their effectiveness in overcoming gradient vanishing and exploding problems; however, compared to conventional RNN, their massive storage and computation requirements hinder their application. In addition, the recurrent structure of RNNs makes them prone to accumulate errors, resulting in a severe loss of accuracy. In this work, we propose a hybrid-iterative compression (HIC) algorithm for LSTM/GRU. By exploiting the error sensitivity of RNN, the gating units are divided into error-sensitive and error-insensitive groups, that are compressed using different algorithms. By using this approach, a 37.1×/32.3× compression ratio is achieved with negligible accuracy loss for LSTM/GRU. Further, an energy efficient accelerator for bidirectional RNNs is proposed. In this accelerator, the data flow of the matrix operation unit based on the block structure matrix (MOU-S) is improved through rearranging weights; the utilization of BRAM is improved through a fine-grained parallelism configuration of matrix-vector multiplications (MVMs). Meanwhile, the timing matching strategy alleviates the load-imbalance problem between MOU-S and the matrix operation unit based on top- k pruning (MOU-P). When running at 200MHz on Xilinx ADM-PCIE-7V3 FPGA, the proposed design achieves an improvement in energy efficiency in a range of 5%-237% for LSTM networks, and an improvement of 58% for GRU networks compared with state-of-the-art designs.

Fiber Clustering Acceleration With a Modified Kmeans++ Algorithm Using Data Parallelism

Fiber clustering methods are typically used in brain research to study the organization of white matter bundles from large diffusion MRI tractography datasets. These methods enable exploratory bundle inspection using visualization and other methods that require identifying brain white matter structures in individuals or a population. Some applications, such as real-time visualization and inter-subject clustering, need fast and high-quality intra-subject clustering algorithms. This work proposes a parallel algorithm using a General Purpose Graphics Processing Unit (GPGPU) for fiber clustering based on the FFClust algorithm. The proposed GPGPU implementation exploits data parallelism using both multicore and GPU fine-grained parallelism present in commodity architectures, including current laptops and desktop computers. Our approach implements all FFClust steps in parallel, improving execution times in all of them. In addition, our parallel approach includes a parallel Kmeans++ algorithm implementation and defines a new variant of Kmeans++ to reduce the impact of choosing outliers as initial centroids. The results show that our approach provides clustering quality results very similar to FFClust, and it requires an execution time of 3.5 s for processing about a million fibers, achieving a speedup of 11.5 times compared to FFClust.

Parallel Large-scale MOSFET Circuit Simulation using Multi-core CPU and Time-saving Techniques

In this paper, we investigate the acceleration properties of fine-grain parallel large-scale MOSFET circuit simulation using multi-core CPU. Several CPU have been tested for understanding maximum possible accelerations. A parallel circuit simulator, MOSTIME, is used to simulate experimental circuits. Performing tests on CPU and MOSTIME, we find that more advanced accelerations on large-scale fine-grain parallel circuit simulation would rely on time-saving software techniques and GPU (Graphic Processing Unit) cards. A newly implemented time-saving technique for MOSTIME has been tested to show promising effects.

Efficient parallel edge-centric approach for relaxed graph pattern matching

Prior algorithms on graph simulation for distributed graphs are not scalable enough as they exhibit heavy message passing. Moreover, they are dependent on the graph partitioning quality that can be a bottleneck due to the natural skew present in real-world data. As a result, their degree of parallelism becomes limited. In this paper, we propose an efficient parallel edge-centric approach for distributed graph pattern matching. We design a novel distributed data structure called ST that allows a fine-grain parallelism, and hence guarantees linear scalability. Based on ST, we develop a parallel graph simulation algorithm called PGSim. Furthermore, we propose PDSim, an edge-centric algorithm that efficiently evaluates dual simulation in parallel. PDSim combines ST and PGSim in a Split-and-Combine approach to accelerate the computation stages. We prove the effectiveness and efficiency of these propositions through theoretical guarantees and extensive experiments on massive graphs. The achieved results confirm that our approach outperforms existing algorithms by more than an order of magnitude.

Efficient exascale discretizations: High-order finite element methods

Efficient exploitation of exascale architectures requires rethinking of the numerical algorithms used in many large-scale applications. These architectures favor algorithms that expose ultra fine-grain parallelism and maximize the ratio of floating point operations to energy intensive data movement. One of the few viable approaches to achieve high efficiency in the area of PDE discretizations on unstructured grids is to use matrix-free/partially assembled high-order finite element methods, since these methods can increase the accuracy and/or lower the computational time due to reduced data motion. In this paper we provide an overview of the research and development activities in the Center for Efficient Exascale Discretizations (CEED), a co-design center in the Exascale Computing Project that is focused on the development of next-generation discretization software and algorithms to enable a wide range of finite element applications to run efficiently on future hardware. CEED is a research partnership involving more than 30 computational scientists from two US national labs and five universities, including members of the Nek5000, MFEM, MAGMA and PETSc projects. We discuss the CEED co-design activities based on targeted benchmarks, miniapps and discretization libraries and our work on performance optimizations for large-scale GPU architectures. We also provide a broad overview of research and development activities in areas such as unstructured adaptive mesh refinement algorithms, matrix-free linear solvers, high-order data visualization, and list examples of collaborations with several ECP and external applications.