Asynchronous Parallel Research Articles

Application of discrete random forest algorithm in multi-person asynchronous parallel disassembly sequence planning for hydropower station equipment maintenance

Application of discrete random forest algorithm in multi-person asynchronous parallel disassembly sequence planning for hydropower station equipment maintenance

An Energy-Efficient Bayesian Neural Network Implementation Using Stochastic Computing Method.

The robustness of Bayesian neural networks (BNNs) to real-world uncertainties and incompleteness has led to their application in some safety-critical fields. However, evaluating uncertainty during BNN inference requires repeated sampling and feed-forward computing, making them challenging to deploy in low-power or embedded devices. This article proposes the use of stochastic computing (SC) to optimize the hardware performance of BNN inference in terms of energy consumption and hardware utilization. The proposed approach adopts bitstream to represent Gaussian random number and applies it in the inference phase. This allows for the omission of complex transformation computations in the central limit theorem-based Gaussian random number generating (CLT-based GRNG) method and the simplification of multipliers as AND operations. Furthermore, an asynchronous parallel pipeline calculation technique is proposed in computing block to enhance operation speed. Compared with conventional binary radix-based BNN, SC-based BNN (StocBNN) realized by FPGA with 128-bit bitstream consumes much less energy consumption and hardware resources with less than 0.1% accuracy decrease when dealing with MNIST/Fashion-MNIST datasets.

Parallel PSO for Efficient Neural Network Training Using GPGPU and Apache Spark in Edge Computing Sets

The training phase of a deep learning neural network (DLNN) is a computationally demanding process, particularly for models comprising multiple layers of intermediate neurons.This paper presents a novel approach to accelerating DLNN training using the particle swarm optimisation (PSO) algorithm, which exploits the GPGPU architecture and the Apache Spark analytics engine for large-scale data processing tasks. PSO is a bio-inspired stochastic optimisation method whose objective is to iteratively enhance the solution to a (usually complex) problem by approximating a given objective. The expensive fitness evaluation and updating of particle positions can be supported more effectively by parallel processing. Nevertheless, the parallelisation of an efficient PSO is not a simple process due to the complexity of the computations performed on the swarm of particles and the iterative execution of the algorithm until a solution close to the objective with minimal error is achieved. In this study, two forms of parallelisation have been developed for the PSO algorithm, both of which are designed for execution in a distributed execution environment. The synchronous parallel PSO implementation guarantees consistency but may result in idle time due to global synchronisation. In contrast, the asynchronous parallel PSO approach reduces the necessity for global synchronization, thereby enhancing execution time and making it more appropriate for large datasets and distributed environments such as Apache Spark. The two variants of PSO have been implemented with the objective of distributing the computational load supported by the algorithm across the different executor nodes of the Spark cluster to effectively achieve coarse-grained parallelism. The result is a significant performance improvement over current sequential variants of PSO.

Replica Exchange of Expanded Ensembles: A Generalized Ensemble Approach with Enhanced Flexibility and Parallelizability.

Generalized ensemble methods such as Hamiltonian replica exchange (HREX) and expanded ensemble (EE) have been shown effective in free energy calculations for various contexts, given their ability to circumvent free energy barriers via nonphysical pathways defined by states with different modified Hamiltonians. However, both HREX and EE methods come with drawbacks, such as limited flexibility in parameter specification or the lack of parallelizability for more complicated applications. To address this challenge, we present the method of replica exchange of expanded ensembles (REXEE), which integrates the principles of HREX and EE methods by periodically exchanging coordinates of EE replicas sampling different yet overlapping sets of alchemical states. With the solvation free energy calculation of anthracene and binding free energy calculation of the CB7-10 binding complex, we show that the REXEE method achieves the same level of accuracy in free energy calculations as the HREX and EE methods, while offering enhanced flexibility and parallelizability. Additionally, we examined REXEE simulations with various setups to understand how different exchange frequencies and replica configurations influence the sampling efficiency in the fixed-weight phase and the weight convergence in the weight-updating phase. The REXEE approach can be further extended to support asynchronous parallelization schemes, allowing looser communications between larger numbers of loosely coupled processors such as cloud computing and therefore promising much more scalable and adaptive executions of alchemical free energy calculations. All algorithms for the REXEE method are available in the Python package ensemble_md, which offers an interface for REXEE simulation management without modifying the source code in GROMACS.

Client selection and resource scheduling in reliable federated learning for UAV-assisted vehicular networks

Federated Learning (FL), a promising deep learning paradigm extensively deployed in Vehicular Edge Computing Networks (VECN), allows a distributed approach to train datasets of nodes locally, e.g., for mobile vehicles, and exchanges model parameters to obtain an accurate model without raw data transmission. However, the existence of malicious vehicular nodes as well as the inherent heterogeneity of the vehicles hinders the attainment of accurate models. Moreover, the local model training and model parameter transmission during FL exert a notable energy burden on vehicles constrained in resources. In view of this, we investigate FL client selection and resource management problems in FL-enabled UAV-assisted Vehicular Networks (FLVN). We first devise a novel reputation-based client selection mechanism by integrating both data quality and computation capability metrics to enlist reliable high-performance vehicles. Further, to fortify the FL reliability, we adopt the consortium blockchain to oversee the reputation information, which boasts tamper-proof and interference-resistant qualities. Finally, we formulate the resource scheduling problem by jointly optimizing the computation capability, the transmission power, and the number of local training rounds, aiming to minimize the cost of clients while guaranteeing accuracy. To this end, we propose a reinforcement learning algorithm employing an asynchronous parallel network structure to achieve an optimized scheduling strategy. Simulation results show that our proposed client selection mechanism and scheduling algorithm can realize reliable FL with an accuracy of 0.96 and consistently outperform the baselines in terms of delay and energy consumption.

The application of chessboard game based on integrated learning and UCT algorithm in mental health and emotional regulation

Go plays a very prominent role in quality education and mental health education. Go can cultivate character, make friends, improve interpersonal relationships, and promote mental health. To apply it to mental health and emotional regulation, the AlphaGo series algorithms are analyzed. To address the unstable deep reinforcement learning training, low training data utilization, model sensitivity to hyper-parameters, and heavy parameter tuning burden, the algorithm is optimized by combining ensemble learning and advanced deep reinforcement learning. Then, the weighted voting method is applied to the selection step of sub nodes in the Monte Carlo tree search process. At the same time, asynchronous parallelization improvements are implemented to improve the efficiency of the Monte Carlo tree search algorithm. Finally, a board game based on ensemble learning and improved Monte Carlo tree search algorithm is obtained. Through experimental analysis, the average winning rate of the system was 93.36 %. The number of users who were very satisfied with the game was 83.96 %. After intervention, the depression and anxiety scores of the experimental group students were significantly lower than the control group (P < 0.05). The designed game can effectively alleviate user anxiety and depression, providing new ideas for the psychotherapy.

Parallel and Distributed Graph Neural Networks: An In-Depth Concurrency Analysis.

Graph neural networks (GNNs) are among the most powerful tools in deep learning. They routinely solve complex problems on unstructured networks, such as node classification, graph classification, or link prediction, with high accuracy. However, both inference and training of GNNs are complex, and they uniquely combine the features of irregular graph processing with dense and regular computations. This complexity makes it very challenging to execute GNNs efficiently on modern massively parallel architectures. To alleviate this, we first design a taxonomy of parallelism in GNNs, considering data and model parallelism, and different forms of pipelining. Then, we use this taxonomy to investigate the amount of parallelism in numerous GNN models, GNN-driven machine learning tasks, software frameworks, or hardware accelerators. We use the work-depth model, and we also assess communication volume and synchronization. We specifically focus on the sparsity/density of the associated tensors, in order to understand how to effectively apply techniques such as vectorization. We also formally analyze GNN pipelining, and we generalize the established Message-Passing class of GNN models to cover arbitrary pipeline depths, facilitating future optimizations. Finally, we investigate different forms of asynchronicity, navigating the path for future asynchronous parallel GNN pipelines. The outcomes of our analysis are synthesized in a set of insights that help to maximize GNN performance, and a comprehensive list of challenges and opportunities for further research into efficient GNN computations. Our work will help to advance the design of future GNNs.

Imperative Process Algebra and Models of Parallel Computation

Studies of issues related to computability and computational complexity involve the use of a model of computation. Central in such a model are computational processes. Processes of this kind can be described using an imperative process algebra based on ACP (Algebra of Communicating Processes). In this paper, it is investigated whether the imperative process algebra concerned can play a role in the field of models of computation. It is demonstrated that the process algebra is suitable to describe in a mathematically precise way models of computation corresponding to existing models based on sequential, asynchronous parallel, and synchronous parallel random access machines as well as time and work complexity measures for those models.

Analyzing cavity evolution and motion characteristics of asynchronous parallel oblique water-entry super-cavitating projectile

Based on the volume of fluid multiphase flow model and the overset mesh technique, a numerical method for an asynchronous parallel oblique water-entry super-cavitating projectile was established. Experimental studies of the oblique water-entry of a high-speed single-launch projectile were carried out to validate the viability of the numerical method. The paper performed the numerical simulations and analyses of cavity evolution and motion characteristics of the front and rear projectiles in different initial intervals and in two sequences of top-side water-entry projectile first and bottom-side water-entry projectile first. The results show that when the initial interval of the first launch projectile is 0.5 time the projectile length, the first launch projectile cannot produce a cavity to completely encapsulate the projectile due to the violent squeezing of the following launch projectile cavity, and its movement is seriously affected and eventually loses its trajectory stability. At the same time, the first launch projectile that enters water from top side is squeezed to a larger degree than the one from bottom side, and the wetting phenomenon occurs earlier and loses stability faster. As the initial interval increases, the influence of the following launch projectile cavity near the first launch projectile is weakened, and the first launch projectile in both water entry sequences move steadily. For the following launch projectile, due to the continuous influence of the first launch projectile cavity, its cavity is always asymmetrical, and its motion stability is affected. The following launch projectile deflects to the inner side and destabilizes when the initial interval is 0.5 times the projectile length. When the initial interval is 1 time the projectile length, it moves steadily. It deflects to the outer side and destabilizes when the initial interval is 2 and 3 times the projectile length. In addition, the motion characteristics of the following launch projectile are basically identical in two water-entry sequences.

HASP: Hierarchical Asynchronous Parallelism for Multi-NN Tasks

HASP: Hierarchical Asynchronous Parallelism for Multi-NN Tasks

Asynchronous Parallel Fuzzy Stochastic Gradient Descent for High-Dimensional Incomplete Data Representation

Asynchronous Parallel Fuzzy Stochastic Gradient Descent for High-Dimensional Incomplete Data Representation

An Asynchronous Parallel I/O Framework for Mass Conservation Ocean Model

I/O is often a performance bottleneck in global ocean circulation models with fine spatial resolution. In this paper, we present an asynchronous parallel I/O framework and demonstrate its efficacy in the Mass Conservation Ocean Model (MaCOM) as a case study. By largely reducing I/O operations in computing processes and overlapping output in I/O processes with computation in computing processes, this framework significantly improves the performance of the MaCOM. Through both reordering output data for maintaining data continuity and combining file access for reducing file operations, the I/O optimizing algorithms are provided to improve output bandwidth. In the case study of the MaCOM, the cost of output in I/O processes can be overlapped by up to 99% with computation in computing processes as decreasing output frequency. The 1D data output bandwidth with these I/O optimizing algorithms is 3.1 times faster than before optimization at 16 I/O worker processes. Compared to the synchronous parallel I/O framework, the overall performance of MaCOM is improved by 38.8% at 1024 computing processes for a 7-day global ocean forecast with 1 output every 2 h through the asynchronous parallel I/O framework presented in this paper.

APapo: An asynchronous parallel optimization method for DNN models

To address the challenges related to segmentation complexity, high memory usage, extended training duration, and low equipment utilization in parallel optimization of large-scale deep neural network (DNN) models, this paper proposes an asynchronous parallel optimization method APapo. Firstly, a multi-iteration asynchronous pipeline parallel scheduling was established for model parallel computing tasks, controlling the specific scheduling process of micro-batch units to address gradient delay updating during asynchronous iteration. Secondly, combined with the given network model and hardware configuration, a dynamic programming strategy for computing resources and model tasks was designed to achieve dynamic segmentation of model computing tasks and optimal matching of computing resources. Finally, an optimization strategy for runtime scheduling of computing resources and model tasks was developed, using improved device streams to maximize the overlap between computing and communication, thus improving the utilization rate of computing resources and reducing training time. Experimental results show that the APapo method achieves fine-grained task segmentation, maximizes the utilization rate of each GPU computing resource, and on average improves the training speed of large-scale deep neural network models by 2.8 times while maintaining the training accuracy of the model compared to existing parallel optimization methods.

A parallel strategy to accelerate neighborhood operation for raster data coordinating CPU and GPU

ABSTRACT This study presents an asynchronous parallel strategy coordinating central processing unit (CPU) and graphic processing unit (GPU) to accelerate neighborhood operation (NO). Specifically, we propose a data partitioning method called multi-anchor task queuing and a task scheduling method called bi-direction task scheduling, which can support CPU and GPU to find the responsible data blocks rapidly and concurrently handle their tasks via a bi-direction merge. Moreover, we optimize the organization of threads distributed among the CPU and GPU. Experimental results show that when a 1.7 GB raster dataset is processed, the speedup ratio achieved by the proposed parallel algorithm reaches 29.63, which is 19% and 18% higher than those of the GPU and standard asynchronous parallel algorithm, respectively. Additionally, the load balance index is below 0.085, which is significantly better than the value achieved by a conventional algorithm. Thus, the strategy achieves a higher speedup ratio and more adaptable load balance, thereby accelerating the NO more efficiently. Further, the impacts of the data volume, computational intensity, organization mode of the GPU threads, and granularity of the GPU stream on the parallel efficiency are evaluated and discussed. We also test the efficiency of four other common NOs with our strategy.

An asynchronous parallel benders decomposition method for stochastic network design problems

Benders decomposition (BD) is one of the most popular solution algorithms for stochastic integer programs. The BD method decomposes stochastic problems into one master problem and multiple disjoint subproblems. It thus lends itself readily to parallelization. In almost all studies on the parallelization of this algorithm, the master problem remains idle until every subproblem is solved and vice versa. This can clearly result in an extremely inefficient parallel algorithm due to excessive idle times. On the other hand, relaxing the synchronization requirement can yield a nonconvergent or less efficient algorithm that may even underperform when compared to the sequential version. Addressing these issues, we introduce an asynchronous parallel BD method for stochastic network design problems. We show that the proposed algorithm converges to the global optimum and suggest various acceleration strategies to enhance its performance. We conduct an extensive numerical study on benchmark instances from a generic stochastic network design problem. The results indicate that using up to 20 processors, our asynchronous algorithm is on average 1.4 times faster than the conventional low-level parallel methods.

Research on the Application and Performance Optimization of GPU Parallel Computing in Concrete Temperature Control Simulation

With the development of engineering technology, engineering has higher requirements for the accuracy and the scale of simulation calculation. The computational efficiency of traditional serial programs cannot meet the requirements of engineering. Therefore, reducing the calculation time of the temperature control simulation program has important engineering significance for real-time simulation of temperature field and stress field, and then adopting more reasonable temperature control and crack prevention measures. GPU parallel computing is introduced into the temperature control simulation program of massive concrete to solve this problem and the optimization is carried out. Considering factors such as GPU clock rate, number of cores, parallel overhead and Parallel Region, the improved GPU parallel algorithm analysis indicator formula is proposed. It makes up for the shortcomings of traditional formulas that focus only on time. According to this formula, when there are enough threads, the parallel effect is limited by the size of the parallel domain, and when the parallel domain is large enough, the efficiency is limited by the parallel overhead and the clock rate. This paper studies the optimal Kernel execution configuration. Shared memory is utilized to improve memory access efficiency by 155%. After solving the problem of bank conflicts, an accelerate rate of 437.5× was realized in the subroutine of the matrix transpose of the solver. The asynchronous parallel of data access and logical operation is realized on GPU by using CUDA Stream, which can overlap part of the data access time. On the basis of GPU parallelism, asynchronous parallelism can double the computing efficiency. Compared with the serial program, the accelerate rate of inner product matrix multiplication of the GPU asynchronous parallel program is 61.42×. This study further proposed a theoretical formula of data access overlap rate to guide the selection of the number of CUDA streams to achieve the optimal computing conditions. The GPU parallel program compiled and optimized by the CUDA Fortran platform can effectively improve the computational efficiency of the simulation program for concrete temperature control, and better serve engineering computing.

Asynchronous Parallel Incremental Block-Coordinate Descent for Decentralized Machine Learning

Machine learning (ML) is a key technique for big-data-driven modelling and analysis of massive Internet of Things (IoT) based intelligent and ubiquitous computing. For fast-increasing applications and data amounts, distributed learning is a promising emerging paradigm since it is often impractical or inefficient to share/aggregate data to a centralized location from distinct ones. This paper studies the problem of training an ML model over decentralized systems, where data are distributed over many user devices and the learning algorithm run on-device, with the aim of relaxing the burden at a central entity/server. Although gossip-based approaches have been used for this purpose in different use cases, they suffer from high communication costs, especially when the number of devices is large. To mitigate this, incremental-based methods are proposed. We first introduce incremental block-coordinate descent (I-BCD) for the decentralized ML, which can reduce communication costs at the expense of running time. To accelerate the convergence speed, an asynchronous parallel incremental BCD (API-BCD) method is proposed, where multiple devices/agents are active in an asynchronous fashion. We derive convergence properties for the proposed methods. Simulation results also show that our API-BCD method outperforms state of the art in terms of running time and communication costs.

Efficient multi-objective optimization through parallel surrogate-assisted local search with tabu mechanism and asynchronous option

Multi-objective optimization is challenging for computationally expensive objectives because most optimization methods rely on a large number of function evaluations to approach the Pareto-optimal front. To this end, this article develops a novel multi-objective optimization algorithm, the Multi-Objective Parallel Local Surrogate-assisted search (MOPLS) algorithm, which combines surrogate approximation and parallel computing. In each iteration, MOPLS incorporates a tabu mechanism to determine new points for expensive evaluations via a series of independent surrogate-assisted local searches. A master–worker architecture in MOPLS allows the algorithm to conduct either synchronous or asynchronous parallel processing. On a number of benchmark problems, MOPLS outperforms recent surrogate-assisted algorithms within a limited computational budget. Empirical results from parallel experiments indicate that MOPLS can significantly improve its efficiency through parallelism, and the asynchronous MOPLS shows advantages in handling objectives with non-constant evaluation times. Finally, the better performance of MOPLS over alternatives in calibrating a complex watershed simulation model demonstrates its competitiveness in solving real-world engineering applications.

A hybrid parallelization approach based on workers grouping algorithm

As the volume of model data increases, traditional machine learning is not able to train models efficiently, so distributed machine learning is gradually used in large-scale data training. Currently, commonly used distributed machine learning algorithms are based on data parallelism, and often use an overall synchronous parallel strategy when passing data, but using this strategy makes the overall training speed limited by the computation speed of the slower workers in the cluster. While the asynchronous parallel strategy maximizes the computational speed of the cluster, there is a delay in updating the parameters of the global model, which may lead to excessive computational errors or non-convergence of the model. In this paper, the author combines these two data delivery methods by grouping workers together and using synchronous parallelism for the workers in the group and asynchronous parallelism for the components for training. The experiment shows that the hybrid parallelism strategy can reduce the training time with guaranteed correctness.

Correction: An asynchronous parallel high-throughput model calibration framework for crystal plasticity finite element constitutive models

Correction: An asynchronous parallel high-throughput model calibration framework for crystal plasticity finite element constitutive models