Synchronous Parallel Research Articles

TAPP: DNN Training for Task Allocation through Pipeline Parallelism Based on Distributed Deep Reinforcement Learning

The rapid development of artificial intelligence technology has made deep neural networks (DNNs) widely used in various fields. DNNs have been continuously growing in order to improve the accuracy and quality of the models. Moreover, traditional data/model parallelism is hard to expand due to communication bottlenecks and hardware efficiency issues. However, pipeline parallelism trains multiple batches, reducing training overheads, so that it can achieve better acceleration effect. Considering the complexity of solving the pipeline parallel task allocation problem in heterogeneous computing resources, in this paper, a task allocation in pipeline parallelism (TAPP) based on deep reinforcement learning, is proposed. In TAPP, the predictive network is trained by a policy gradient until it obtains the optimal pipeline parallel task allocation scheme and speeds up the model training. Experimental results show that, on average, the single-step training time of TAPP is decreased by 1.37 times and the proportion of communication time is reduced by 48.92%, compared with the data parallelism, bulk synchronous parallel (BSP).

Model averaging in distributed machine learning: a case study with Apache Spark

The increasing popularity of Apache Spark has attracted many users to put their data into its ecosystem. On the other hand, it has been witnessed in the literature that Spark is slow when it comes to distributed machine learning (ML). One resort is to switch to specialized systems such as parameter servers, which are claimed to have better performance. Nonetheless, users have to undergo the painful procedure of moving data into and out of Spark. In this paper, we investigate performance bottlenecks of MLlib (an official Spark package for ML) in detail, by focusing on analyzing its implementation of stochastic gradient descent (SGD)—the workhorse under the training of many ML models. We show that the performance inferiority of Spark is caused by implementation issues rather than fundamental flaws of the bulk synchronous parallel (BSP) model that governs Spark’s execution: we can significantly improve Spark’s performance by leveraging the well-known “model averaging” (MA) technique in distributed ML. Indeed, model averaging is not limited to SGD, and we further showcase an application of MA to training latent Dirichlet allocation (LDA) models within Spark. Our implementation is not intrusive and requires light development effort. Experimental evaluation results reveal that the MA-based versions of SGD and LDA can be orders of magnitude faster compared to their counterparts without using MA.

Stochastic gradient descent‐based support vector machines training optimization on Big Data and HPC frameworks

SummarySupport vector machines (SVM) is a widely used machine learning algorithm. With the increasing amount of research data nowadays, understanding how to do efficient training is more important than ever. This article discusses the performance optimizations and benchmarks related to providing high‐performance support for SVM training. In this research, we have focused on a highly scalable gradient descent‐based approach to implementing the core SVM algorithm. In providing a scalable solution, we have designed optimized high‐performance computing and dataflow‐oriented SVM implementations. A high‐performance computing approach means the algorithm is implemented with the bulk synchronous parallel (BSP) model. In addition, we analyzed the language level optimizations and math kernel optimizations on a prominent HPC modeling programming language (C++) and dataflow modeling programming language (Java). In the experiments, we compared the performance of classic HPC models, classic dataflow models, and hybrid models designed on classic HPC and dataflow programming models. Our research illustrates a scientific approach in designing the SVM algorithm at scale in classic HPC, dataflow, and hybrid systems.

Efficient Communication Scheduling for Parameter Synchronization of DML in Data Center Networks

It's common practice to speed up machine learning (ML) training by distributing it across a cluster of computing nodes. Data-parallel distributed ML (DML) training relieves the pressure of computing node; however, the communication traffic introduced during parameter synchronization becomes bottleneck of DML training. Regarding communication bottleneck, we identify two primary causes: high contention of concurrent communication and large volume of redundant transmission in the push and pull stage while synchronizing parameters. To address these issues, we propose a novel Group Stale Synchronous Parallel (GSSP) scheme, which divides the nodes into groups and coordinates groups to synchronize in a circular order. GSSP mitigates network contention and is proven to converge. We provide analysis of the optimal group's number based on bandwidth and buffer size. For reducing traffic redundancy, we propose a multicast-based scheme, which generates multicast trees by minimizing links overlap and allocates transmission rate to multicast flows by solving the min-max optimization problem. Finally, we conduct extensive simulations to evaluate the performance of our proposals. We simulate parameter transmission of All-Reduce and parameter server in Fat-Tree with traffics trace of ML models. Simulation results show that our proposals provide communication-efficiency for DML training by mitigating contention and reducing redundancy.

GSSP: Eliminating Stragglers Through Grouping Synchronous for Distributed Deep Learning in Heterogeneous Cluster

Distributed deep learning has been widely used in training deep neural networks, especially for big models on massive datasets. Parameter Server (PS) architecture is the most popular distributed training framework, which can flexibly design the global parameter update manner. However, when scaling to complex heterogeneous clusters, stragglers make it difficult for existing distributed paradigms on PS framework to balance between synchronous waiting and staleness, which slows down the model training sharply. In this article, we propose <i>Grouping Stale Synchronous Parallel</i> (GSSP) scheme, which groups workers with similar performance together. Group servers coordinate intra-group workers using Stale Synchronous Parallel while they communicate with each other asynchronously to eliminate stragglers and refine the model weights. We further propose <i>Grouping Dynamic Tok-K Sparsification</i> (GDTopK), which dynamically adjusts the upload ratio for each group so as to make communication volume differentiated and mitigate inter-group iteration speed gap. We have conducted experiments on LeNet-5 on MNIST, ResNet-18, VGG-19 on Cifar-10, and Seq2Seq on Multi30k. Results show that GSSP accelerates the training by 46% <inline-formula><tex-math notation="LaTeX">$\sim$</tex-math></inline-formula> 120%, with less than 1 percent accuracy drop. And GDTopK can make up for part of the lost accuracy.

Development of Art and Culture Creative Industry Using FPGA and Dynamic Image Sampling

The TVPS146’s digital video processor is used for digital video processing and image output on digital signal processing (DSP) ports. The following is a summary of the field programmable gate array (FPGA) features and programming principles. FPGA is designed by the ping‐pong operation, including serial to parallel conversion design, pipeline operation, and data interface synchronization. A dance video screenshot analysis microsystem is composed of it and video dynamic image processing. Finally, according to the number of dance culture performance institutions based on the internet from 2014 to 2018, the development strategy of dance culture and creativity based on the internet is put forward. The results show that the image sharpness of the video screenshots processed by the system is improved, and the effect of enhanced processing is achieved compared to the unprocessed video screenshots. In addition, after the video screenshot processed by the system is compressed, some details are lost and the outline of the image is blurred. The combination of internet immediacy, mass, grassroots, and interactivity can accelerate the development of China’s dance culture and creative industry and provide an effective practical foundation for innovation in the development of art and culture.

Performance Comparison of Three Hybridization Categories to Solve Multi-Objective Flow Shop Scheduling Problem

The industries must preserve a rate of constant productivity; however, weaknesses appear at the level of production system which engenders high manufacturing costs. Scheduling is considered the most significant issue in the production system, the solution to that problem need complex methods to solve it. The goal of this paper is to establish three hybridization categories of the evolutionary methods ABC and PSO to solve multi-objective flow shop scheduling problem: Synchronous parallel hybridization using the weighted sum method of the fitness function, sequential hybridization using or not using the weighted sum method of the fitness function, and asynchronous parallel hybridization using the weighted sum method of the fitness function. Then to test these methods in an automotive multi-objective flow shop and to perform an in-depth comparison for verifying how the multi hybridization and the hybridization categories influence the resolution of multi-objective flow shop scheduling problems. The results are consistent with other studies that have shown that the multi hybridization improve the effectiveness of the algorithm.

Binocular stereo matching algorithm based on MST cost aggregation.

For common binocular stereo matching algorithms in computer vision, it is not easy to obtain high precision and high matching speed at the same time. In this paper, an improved binocular stereo matching algorithm based on Minimum Spanning Tree (MST) cost aggregation is proposed. Firstly, the performance of the parallel algorithm can be improved by reducing the height of the tree. Then, an improved Root to Leaf (L2R) cost aggregation algorithm is proposed. By combining stereo matching technology with parallel computing technology, the above method can realize synchronous parallel computing at the algorithm level. Experimental results show that the improved algorithm has high accuracy and high matching speed for binocular stereo vision.

Chaos-control and parallel queue synchronization of laser local area network

In this work, we study the chaos-control and parallel queue synchronization of a laser local area network (LAN). We present and study specifically a “single-queue-double-parameter” method of the parallel series queue dynamic behavior synchronization of the controlled laser LAN under two optoelectronic delay feedback controllers, and establish the mathematical and physical model of the controlled laser LAN. The LAN node is composed of two space coupled lasers with different parameters and other two single lasers, where two lasers series produce two different parallel queues, which results in two different chains of LAN nodes. Optical LAN links are composed of two optical parallel-crossing paths and two photoelectric delay feedback controllers setting in two lasers of LAN, which creates a method of double-parameter control of LAN. Through the analysis of the stability theory of differential equation and the dynamic characteristic equation of coupled lasers of LAN, our mathematical theory demonstrates that the chaos-control of laser LAN can be achieved by two photoelectric delay feedback controllers adjusting photoelectric feedback levels and feedback delay time of one of the two coupled laser and another single laser, respectively. Making analysis of the stability theory of differential equation and the dynamic characteristic equation of LAN nodes in two queue chains, we demonstrate theoretically how to obtain synchronization in network nodes of the controlled LAN on two queue chains by controlling optical feedback levels, and by the photoelectric delay feedback controllers adjusting photoelectric feedback levels and feedback delay time, respectively. Using our numerical calculation of parallel queue synchronization, the node laser’s waveform and its phase space trajectory, we find that very lasers of network nodes of the controlled LAN can lead to the parallel queue synchronization of a double-period, a three-period, a four-period and other quasi-periods while these quasi-periodic synchronizations and dynamic synchronizations are controlled in two queue chains of LAN nodes when we let the photoelectric feedback level and the delay time shift on some parameters. We find also two controlled quasi-periodic parallel queue synchronization regions. This paper also presents an application case of laser LAN multi-point chaotic carrier synchronous emission and ultra-wideband communication. This is a new type of controlled laser LAN system, which has the core elements of optical LAN and the characteristics of multi-variable, multi-dimension and parallel queue chaos-control techniques of complex dynamic networks. It also has the function of optical network ultra-wideband communication. The results have important reference value for studying the LAN, optical network and its synchronization and control, laser technology and chaos.

Petrel: Heterogeneity-Aware Distributed Deep Learning Via Hybrid Synchronization

The parameter server (PS) paradigm has achieved great success in deploying large-scale distributed Deep Learning (DL) systems. However, these systems implicitly assume that the cluster is homogeneous and this assumption does not hold in many realworld cases. Although the previous efforts are paid to address heterogeneity, they mainly prioritize the contribution of fast workers and reduce the involvement of slow workers, resulting in the limitations of workload imbalance and computation inefficiency. We reveal that grouping workers into communities, an abstraction proposed by us, and handling parameter synchronization at the community level can conquer these limitations and accelerate the training convergence progress. The inspiration of community comes from our exploration of prior knowledge about the similarity between workers, which is often neglected by previous work. These observations motivate us to propose a new synchronization mechanism named Community-aware Synchronous Parallel (CASP), which uses the Asynchronous Advantage Actor-Critic (A3C)-based algorithm to intelligently determine community configuration and fully improve the synchronization performance. The whole idea has been implemented in a prototype system called Petrel that achieves a good balance between convergence efficiency and communication overhead. The evaluation under various benchmarks with multiple metrics and baseline comparison demonstrates the effectiveness of Petrel. Specifically, Petrel accelerates the training convergence speed by up to 1.87 x faster and reduces communication traffic by up to 26.85 percent, on average, over the non-community synchronization mechanisms.

Synchronous and asynchronous parallel computation for large-scale optimal control of connected vehicles

Connected vehicles is an important intelligent transportation system to improve the traffic performance. This paper proposes two parallel computation algorithms to solve a large-scale optimal control problem in the coordination of multiple connected vehicles. The coordination is formulated as a centralized optimization problem in the receding horizon fashion. A decentralized computation network is designed to facilitate the development of parallel algorithms. We use Taylor series to linearize non-convex constraints, and introduce a set of consensus constraints to transform the centralized problem to a standard consensus optimization problem. A synchronous parallel algorithm is firstly proposed to solve the consensus optimization problem by applying the alternating direction method of multipliers (ADMM). The ADMM framework allows us to decompose the coupling constraints and decision variables, leading to parallel iterations for each vehicle in a synchronous fashion. We then propose an asynchronous version of the parallel algorithm that allows the vehicles to update their variables asynchronously in the computation network. The effectiveness and efficiency of the proposed algorithms are validated by extensive numerical simulations.

Comparison of synchronous and asynchronous parallelization of extreme surrogate-assisted multi-objective evolutionary algorithm

This paper investigates the integration of a surrogate-assisted multi-objective evolutionary algorithm (MOEA) and a parallel computation scheme to reduce the computing time until obtaining the optimal solutions in evolutionary algorithms (EAs). A surrogate-assisted MOEA solves multi-objective optimization problems while estimating the evaluation of solutions with a surrogate function. A surrogate function is produced by a machine learning model. This paper uses an extreme learning surrogate-assisted MOEA/D (ELMOEA/D), which utilizes one of the well-known MOEA algorithms, MOEA/D, and a machine learning technique, extreme learning machine (ELM). A parallelization of MOEA, on the other hand, evaluates solutions in parallel on multiple computing nodes to accelerate the optimization process. We consider a synchronous and an asynchronous parallel MOEA as a master-slave parallelization scheme for ELMOEA/D. We carry out an experiment with multi-objective optimization problems to compare the synchronous parallel ELMOEA/D with the asynchronous parallel ELMOEA/D. In the experiment, we simulate two settings of the evaluation time of solutions. One determines the evaluation time of solutions by the normal distribution with different variances. On the other hand, another evaluation time correlates to the objective function value. We compare the quality of solutions obtained by the parallel ELMOEA/D variants within a particular computing time. The experimental results show that the parallelization of ELMOEA/D significantly reduces the computational time. In addition, the integration of ELMOEA/D with the asynchronous parallelization scheme obtains higher quality of solutions quicker than the synchronous parallel ELMOEA/D.

Performance prediction for distributed graph computing

SummaryPerformance prediction for executing graph applications on distributed systems is a prerequisite to improve system performance. Especially for distributed systems optimized by sacrificing the accuracy of results to improve runtime performance, performance prediction can be used to determine accuracy‐related system parameters to achieve a tradeoff between the runtime and inaccuracy of results. This article presents an approach that predicts the runtime and inaccuracy of executing graph algorithms on bulk synchronous parallel (BSP)‐based systems to optimize system parameters by using an artificial neural network (ANN). The proposed approach samples different scales of subgraphs from the input graph by maintaining that the features of subgraphs are similar to the features of the input graph. Then it executes graph algorithm on each subgraph and extracts their runtime features. An ANN‐based performance prediction model is trained off‐line based on the extracted features and is used to predict the performance of executing graph algorithm on the complete input graph. We have validated the proposed approach by conducting single‐source shortest path, connected component, and PageRank on the BSP‐based distributed systems. The experimental results demonstrate that the prediction method can effectively predict the runtime and the inaccuracy with a relative error rate under 14% and under 25%, respectively, compared with the actual performance results.

Adaptive Asynchronous Parallelization of Graph Algorithms

This article proposes an Adaptive Asynchronous Parallel (AAP) model for graph computations. As opposed to Bulk Synchronous Parallel (BSP) and Asynchronous Parallel (AP) models, AAP reduces both stragglers and stale computations by dynamically adjusting relative progress of workers. We show that BSP, AP, and Stale Synchronous Parallel model (SSP) are special cases of AAP. Better yet, AAP optimizes parallel processing by adaptively switching among these models at different stages of a single execution. Moreover, employing the programming model of GRAPE, AAP aims to parallelize existing sequential algorithms based on simultaneous fixpoint computation with partial and incremental evaluation. Under a monotone condition, AAP guarantees to converge at correct answers if the sequential algorithms are correct. Furthermore, we show that AAP can optimally simulate MapReduce, PRAM, BSP, AP, and SSP. Using real-life and synthetic graphs, we experimentally verify that AAP outperforms BSP, AP, and SSP for a variety of graph computations.

Synthesis of Parallel Synchronous Software

In typical embedded applications, the precise execution time of the program does not matter and it is sufficient to meet a real-time deadline. However, modern applications in information security have become much more time-sensitive due to the risk of timing side-channel leakage. The timing of such programs needs to be data-independent and precise. We describe a parallel synchronous software model, which executes as $N$ parallel threads on a processor with word-length $N$ . Each thread is a single-bit synchronous machine with precise, contention-free timing, while each of the $N$ threads still executes as an independent machine. The resulting software supports fine-grained parallel execution. In contrast to earlier work to obtain precise and repeatable timing in software, our solution does not require modifications to the processor architecture nor specialized instruction scheduling techniques. In addition, all threads run in parallel and without contention, which eliminates the problem of thread scheduling. We use hardware (HDL) semantics to describe a thread as a single-bit synchronous machine. Using logic synthesis and code generation, we derive a parallel synchronous implementation of this design. We illustrate the synchronous parallel programming model with practical examples from cryptography and other applications with precise timing requirements.

BaPa: A Novel Approach of Improving Load Balance in Parallel Matrix Factorization for Recommender Systems

A simplified approach to accelerate matrix factorization of big data is to parallelize it. A commonly used method is to divide the matrix into multiple non-intersecting blocks and concurrently calculate them. This operation causes the Load balance problem, which significantly impacts parallel performance and is a big concern. A general belief is that the load balance across blocks is impossible by balancing rows and columns separately. We challenge the belief by proposing an approach of “Balanced Partitioning (BaPa)”. We demonstrate under what circumstance independently balancing rows and columns can lead to the balanced intersection of rows and columns, why, and how. We formally prove the feasibility of BaPa by observing the variance of rating numbers across blocks, and empirically validate its soundness by applying it to two standard parallel matrix factorization algorithms, DSGD and CCD++. Besides, we establish a mathematical model of “Imbalance Degree” to explain further why BaPa works well. BaPa is applied to synchronous parallel matrix factorization, but as a general load balance solution, it has significant application potential.

Distributed Graph Computation Meets Machine Learning

TuX2 is a new distributed graph engine that bridges graph computation and distributed machine learning. TuX2 inherits the benefits of elegant graph computation model, efficient graph layout, and balanced parallelism to scale to billion-edge graphs, while extended and optimized for distributed machine learning to support heterogeneity in data model, Stale Synchronous Parallel in scheduling, and a new Mini-batch, Exchange, GlobalSync, and Apply ( MEGA ) model for programming. TuX2 further introduces a hybrid vertex-cut graph optimization and supports various consistency models in fault tolerance for machine learning. We have developed a set of representative distributed machine learning algorithms in TuX2 , covering both supervised and unsupervised learning. Compared to the implementations on distributed machine learning platforms, writing those algorithms in TuX2 takes only about 25 percent of the code: our graph computation model hides the detailed management of data layout, partitioning, and parallelism from developers. The extensive evaluation of TuX2 , using large datasets with up to 64 billion of edges, shows that TuX2 outperforms PowerGraph/PowerLyra, the state-of-the-art distributed graph engines, by an order of magnitude, while beating two state-of-the-art distributed machine learning systems by at least 60 percent.

Synchronous Parallel Block Coordinate Descent Method for Nonsmooth Convex Function Minimization

This paper proposes a synchronous parallel block coordinate descent algorithm for minimizing a composite function, which consists of a smooth convex function plus a non-smooth but separable convex function. Due to the generalization of the proposed method, some existing synchronous parallel algorithms can be considered as special cases. To tackle high dimensional problems, the authors further develop a randomized variant, which randomly update some blocks of coordinates at each round of computation. Both proposed parallel algorithms are proven to have sub-linear convergence rate under rather mild assumptions. The numerical experiments on solving the large scale regularized logistic regression with l1 norm penalty show that the implementation is quite efficient. The authors conclude with explanation on the observed experimental results and discussion on the potential improvements.

Asynchronous Lagrangian scenario decomposition

We present a distributed asynchronous algorithm for solving two-stage stochastic mixed-integer programs (SMIP) using scenario decomposition, aimed at industrial-scale instances of the stochastic unit commitment (SUC) problem. The algorithm is motivated by large differences in run times observed among scenario subproblems of SUC instances, which can result in inefficient use of distributed computing resources by synchronous parallel algorithms. Our algorithm performs dual iterations asynchronously using a block-coordinate subgradient descent method which allows performing block-coordinate updates using delayed information, while candidate primal solutions are recovered from the solutions of scenario subproblems using heuristics. We present a high performance computing implementation of the asynchronous algorithm, detailing the operations performed by each parallel process and the communication mechanisms among them. We conduct numerical experiments using SUC instances of the Western Electricity Coordinating Council system with up to 1000 scenarios and of the Central Western European system with up to 120 scenarios. We also conduct numerical experiments on generic SMIP instances from the SIPLIB library (DCAP and SSLP). The results demonstrate the general applicability of the proposed algorithm and its ability to solve industrial-scale SUC instances within operationally acceptable time frames. Moreover, we find that an equivalent synchronous parallel algorithm would leave cores idle up to 80.4% of the time on our realistic test instances, an observation which underscores the need for designing asynchronous optimization schemes in order to fully exploit distributed computing on real world applications.

Falcon: Addressing Stragglers in Heterogeneous Parameter Server Via Multiple Parallelism

The parameter server architecture has shown promising performance advantages when handling deep learning (DL) applications. One crucial issue in this regard is the presence of stragglers, which significantly retards DL training progress. Previous solutions for solving stragglers may not fully exploit the computation resource of the cluster as evidenced by our experiments, especially in the heterogeneous environment. This motivates us to design a heterogeneity-aware parameter server paradigm that addresses stragglers and accelerates DL training from the perspective of computation parallelism. We introduce a novel methodology named straggler projection to give a comprehensive inspection of stragglers and reveal practical guidelines to solve this problem in two aspects: (1) controlling each worker's training speed via elastic training parallelism control and (2) transferring blocked tasks from stragglers to pioneers to fully utilize the computation resource. Following these guidelines, we propose the abstraction of parallelism as an infrastructure and design the Elastic-Parallelism Synchronous Parallel (EPSP) algorithm to handle distributed training and parameter synchronization, supporting both enforcedand slack-synchronization schemes. The whole idea has been implemented into a prototype called Falcon which effectively accelerates the DL training speed with the presence of stragglers. Evaluation under various benchmarks with baseline comparison demonstrates the superiority of our system. Specifically, Falcon reduces the training convergence time, by up to 61.83, 55.19, 38.92, and 23.68 percent shorter than FlexRR, Sync-opt, ConSGD, and DynSGD, respectively.