Bulk Synchronous Parallel Model Research Articles

Efficient Edge Computing: A Survey of High-Throughput Concurrent Processing Strategies for Graph Data

This paper reviews the strategies for high-throughput concurrent processing of graph data in edge computing environments. As information technology rapidly advances, particularly in areas such as the Internet of Things (IoT), smart cities, and autonomous driving, the need for real-time and efficient data processing continues to grow. Edge computing is a distributed computing paradigm that can process data at or near its origin, thereby reducing network latency, improving application performance, and reducing the load on central data centers. The discussion includes the application of edge computing in graph data processing, highlighting parallel computing models such as the Bulk Synchronous Parallel (BSP) model and other parallel optimization techniques like data partitioning and task scheduling. The paper also addresses specific challenges of parallel processing in edge computing, such as resource constraints, data communication delays, and strategies for security and privacy protection. Despite the significant potential edge computing has demonstrated in processing graph data, numerous challenges remain. Future research directions involve the development of new resource optimization algorithms, low-latency communication protocols, and technologies to enhance data security, aiming to achieve more efficient and secure graph data processing. This paper provides clear directions and a foundation for the efficient implementation of graph data processing in edge computing environments, positioning edge computing to play an increasingly significant role in real-time data analysis and intelligent applications.

FSM-BC-BSP: Frequent Subgraph Mining Algorithm Based on BC-BSP

As graph models become increasingly prevalent in the processing of scientific data, the exploration of effective methods for the mining of meaningful patterns from large-scale graphs has garnered significant research attention. This paper delves into the complexity of frequent subgraph mining and proposes a frequent subgraph mining (FSM) algorithm. This FSM algorithm is developed within a distributed graph iterative system, designed for the Big Cloud (BC) environment of the China Mobile Corp., and is based on the bulk synchronous parallel (BSP) model, named FSM-BC-BSP. Its aim is to address the challenge of mining frequent subgraphs within a single, large graph. This study advocates for the incorporation of a message sending and receiving mechanism to facilitate data sharing across various stages of the frequent subgraph mining algorithm. Additionally, it suggests employing a standard coded subgraph and sending it to the same node for global support calculation on the large graph. The adoption of the rightmost path expansion strategy in generating candidate subgraphs helps to mitigate the occurrence of redundant subgraphs. The use of standard coding ensures the unique identification of subgraphs, thus eliminating the need for isomorphism calculations. Support calculation is executed using the Minimum Image (MNI) measurement method, aligning with the downward closure attribute. The experimental results demonstrate the robust performance of the FSM-BC-BSP algorithm across diverse input datasets and parameter configurations. Notably, the algorithm exhibits exceptional efficacy, particularly in scenarios with low support requirements, showcasing its superior performance under such conditions.

Dynamic Straggler Mitigation for Large-Scale Spatial Simulations

Spatial simulations have been widely used to study real-world environments, such as transportation systems. Applications like prediction and analysis of transportation require the simulation to handle millions of objects while running faster than real time. Running such large-scale simulation requires high computational power, which can be provided through parallel distributed computing. Implementations of parallel distributed spatial simulations usually follow a bulk synchronous parallel (BSP) model to ensure the correctness of simulation. The processing in BSP is divided into iterations of computation and communication, running on multiple workers, followed by a global barrier synchronisation to ensure that all communications are concluded. Unfortunately, the BSP model is plagued by the straggler problem, where a delay in any worker slows down the entire simulation. Stragglers may occur for many reasons, including imbalanced workload distribution or communication and synchronisation delays. The straggler problem can become more severe with increasing parallelism and continuous change of workload distribution among workers. This article proposes methods to dynamically mitigate stragglers and tackle communication delays. The proposed strategies can rebalance the workload distribution during simulation. These methods employ the spatial properties of the simulated environments to combine a flexible synchronisation model with decentralised dynamic load balancing and on-demand resource allocation. All proposed methods are implemented and evaluated using a microscopic traffic simulator as an example of large-scale spatial simulations. We run traffic simulations for Melbourne, Beijing and New York with different straggler scenarios. Our methods significantly improve simulation performance compared to advanced methods such as global dynamic load balancing.

A Fine-grained Asynchronous Bulk Synchronous parallelism model for PGAS applications

The Partitioned Global Address Space (PGAS) model is well suited for executing irregular applications on cluster-based systems, due to its efficient support for short, one-sided messages. Separately, the actor model has been gaining popularity as a productive asynchronous message-passing approach for distributed objects in enterprise and cloud computing platforms, typically implemented in languages such as Erlang, Scala or Rust. To the best of our knowledge, there has been no past work on using the actor model to deliver both productivity and scalability to irregular PGAS applications with large number of small messages.In this paper, we introduce a new programming system for PGAS applications, in which point-to-point remote operations can be expressed as fine-grained asynchronous actor messages. In our approach, the programmer does not need to worry about programming complexities related to message aggregation and termination detection. Our approach can be viewed as extending the classical Bulk Synchronous Parallelism model with fine-grained asynchronous communications within a phase or superstep. We believe that our approach offers a desirable point in the productivity-performance space for PGAS applications, with more scalable performance and higher productivity relative to past approaches. Specifically, for seven irregular mini-applications from the Bale Kernels and three graph kernels executed using 2048 cores in the NERSC Cori system, our approach shows geometric mean performance improvements of ≥20× relative to standard PGAS versions (UPC and OpenSHMEM) while maintaining comparable productivity to those versions.This is an extended version of the conference paper “A Productive and Scalable Actor-Based Programming System for PGAS Applications” (Paul et al., 2022)[1] from ICCS 2022.

ZipLine: an optimized algorithm for the elastic bulk synchronous parallel model

The bulk synchronous parallel (BSP) is a celebrated synchronization model for general-purpose parallel computing that has successfully been employed for distributed training of deep learning models. A shortcoming of the BSP is that it requires workers to wait for the straggler at every iteration. Therefore, employing BSP increases the waiting time of the faster workers of a cluster and results in an overall prolonged training time. To ameliorate this shortcoming of BSP, we propose ElasticBSP, a model that aims to relax its strict synchronization requirement with an elastic synchronization by allowing delayed synchronization to minimize the waiting time. ElasticBSP offers more flexibility and adaptability during the training phase, without sacrificing the accuracy of the trained model. ElasticBSP is realized by the algorithm named ZipLine, which consists of two phases. First, it estimates for each worker the end time points of its future iterations at run time, and then a one-pass algorithm over the estimated time points of all workers is employed to fast compute an optimal future time point for synchronization. We provide theoretical results about the correctness and performance of the ZipLine algorithm. Furthermore, we propose algorithmic and implementation optimizations of ZipLine, namely ZipLineOpt and ZipLineOptBS, which reduce the time complexity of ZipLine to linearithmic time. A thorough experimental evaluation demonstrates that our proposed ElasticBSP model, materialized by the proposed optimized ZipLine variants, converges faster and to a higher accuracy than the predominant BSP. The focus of the paper is on optimizing the synchronization scheduling over a parameter server architecture. It is orthogonal to other types of optimizations, such as the learning rate optimization.

Model averaging in distributed machine learning: a case study with Apache Spark

The increasing popularity of Apache Spark has attracted many users to put their data into its ecosystem. On the other hand, it has been witnessed in the literature that Spark is slow when it comes to distributed machine learning (ML). One resort is to switch to specialized systems such as parameter servers, which are claimed to have better performance. Nonetheless, users have to undergo the painful procedure of moving data into and out of Spark. In this paper, we investigate performance bottlenecks of MLlib (an official Spark package for ML) in detail, by focusing on analyzing its implementation of stochastic gradient descent (SGD)—the workhorse under the training of many ML models. We show that the performance inferiority of Spark is caused by implementation issues rather than fundamental flaws of the bulk synchronous parallel (BSP) model that governs Spark’s execution: we can significantly improve Spark’s performance by leveraging the well-known “model averaging” (MA) technique in distributed ML. Indeed, model averaging is not limited to SGD, and we further showcase an application of MA to training latent Dirichlet allocation (LDA) models within Spark. Our implementation is not intrusive and requires light development effort. Experimental evaluation results reveal that the MA-based versions of SGD and LDA can be orders of magnitude faster compared to their counterparts without using MA.

Stochastic gradient descent‐based support vector machines training optimization on Big Data and HPC frameworks

SummarySupport vector machines (SVM) is a widely used machine learning algorithm. With the increasing amount of research data nowadays, understanding how to do efficient training is more important than ever. This article discusses the performance optimizations and benchmarks related to providing high‐performance support for SVM training. In this research, we have focused on a highly scalable gradient descent‐based approach to implementing the core SVM algorithm. In providing a scalable solution, we have designed optimized high‐performance computing and dataflow‐oriented SVM implementations. A high‐performance computing approach means the algorithm is implemented with the bulk synchronous parallel (BSP) model. In addition, we analyzed the language level optimizations and math kernel optimizations on a prominent HPC modeling programming language (C++) and dataflow modeling programming language (Java). In the experiments, we compared the performance of classic HPC models, classic dataflow models, and hybrid models designed on classic HPC and dataflow programming models. Our research illustrates a scientific approach in designing the SVM algorithm at scale in classic HPC, dataflow, and hybrid systems.

Adaptive Asynchronous Parallelization of Graph Algorithms

This article proposes an Adaptive Asynchronous Parallel (AAP) model for graph computations. As opposed to Bulk Synchronous Parallel (BSP) and Asynchronous Parallel (AP) models, AAP reduces both stragglers and stale computations by dynamically adjusting relative progress of workers. We show that BSP, AP, and Stale Synchronous Parallel model (SSP) are special cases of AAP. Better yet, AAP optimizes parallel processing by adaptively switching among these models at different stages of a single execution. Moreover, employing the programming model of GRAPE, AAP aims to parallelize existing sequential algorithms based on simultaneous fixpoint computation with partial and incremental evaluation. Under a monotone condition, AAP guarantees to converge at correct answers if the sequential algorithms are correct. Furthermore, we show that AAP can optimally simulate MapReduce, PRAM, BSP, AP, and SSP. Using real-life and synthetic graphs, we experimentally verify that AAP outperforms BSP, AP, and SSP for a variety of graph computations.

Processing graphs with barrierless asynchronous parallel model on shared-memory systems

Graph analytics systems rely on parallel models to realize parallel processing of large-scale graphs, in which the most extensively used models are the bulk synchronous parallel (BSP) and the asynchronous parallel (AP) model. The BSP model is simple and deterministic but has difficulties in expressing some graph processing algorithms, while the AP model is flexible to handle various kinds of algorithms but behaves inefficiently on memory-intensive algorithms due to higher scheduling overhead and internal global barriers. Additionally, although the matrix computation has demonstrated its high-performance characteristic in graph processing, most matrix-based systems only support the BSP model. To improve efficiency of the AP model for graph analytics, this paper proposes a barrierless AP model which uses matrix operations to process graphs and removes the global barrier in each superstep of the AP model to accelerate the execution progress of algorithms. The proposed model is evaluated by comparing it with both AP and BSP-based systems including GraphLab, GRACE, and GraphMat. Experimental results show that our model behaves better than the asynchronous systems, and even outperforms the BSP-based system on some memory-intensive graph algorithms as well as computation-intensive graph algorithms.

A semi-Automatic Approach for Parallel Problem Solving using the Multi-BSP Model

The Multi-Bulk Synchronous Parallel (Multi-BSP) model is a recently proposed parallel programming model for multicore machines that extends the classic Bulk Synchronous Parallel model. Multi-BSP aims to be a useful model to design algorithms and estimate their running time. This model heavily relies on the right computation of parameters that characterize the hardware. Of course, the hardware utilization also depends on the specific features of the problems and the algorithms applied to solve them. This article introduces a semi-automatic approach for solving problems applying parallel algorithms using the Multi-BSP model. First, the specific multicore computer to use is characterized by applying an automatic procedure. After that, the hardware architecture discovered in the previous step is considered in order to design a portable parallel algorithm. Finally, a fine tuning of parameters is performed to improve the overall efficiency. We propose a specific benchmark for measuring the parameters that characterize the communication and synchronization costs in a particular hardware. Our approach discovers the hierarchical structure of the multicore architecture and compute both parameters for each level that can share data and make synchronizations between computing units. A second contribution of our research is a proposal for a Multi-BSP engine. It allows designing algorithms by applying a recursive methodology over the hierarchical tree already built by the benchmark, focusing on three atomic functions based in a divide-and-conquer strategy. The validation of the proposed method is reported, by studying an algorithm implemented in a prototype of the Multi-BSP engine, testing different parameter configurations that best fit to each problem and using three different high-performance multicore computers.

BSP abstract state machines capture bulk synchronous parallel computations

The bulk synchronous parallel (BSP) bridging model is a model for concurrent computations with alternating computation and communication phases between programs running on different processors. In a computation phase the programs proceed independently and asynchronously until a barrier is reached. In a communication phase initiated by all programs having reached the barrier only messages between the programs are exchanged until normal processing can be continued. In this article we present a behavioural theory for BSP computations comprising an axiomatisation of the BSP model, the definition of a restricted class of concurrent abstract state machines, which we call BSP abstract state machines, and the proof that BSP abstract state machines capture BSP computations as defined by the axiomatisation. We illustrate the use of BSP abstract state machines on map-reduce. • Language-independent axiomatisation of the BSP bridging model. • Proof that only from the axiomatisation the alternation of computation and communication phases can be obtained. • Definition of BSP-ASMs. • Proof that BSP-ASMs capture BSP algorithms as defined by the axiomatisation.

PCTL model checking based on Giraph

At present, ensuring system security and verifying whether the system meets the given requirements have been the focus of computer engineers. Model checking is an automatic and exhaustive technique to prove whether there has absence of such errors in a system which has been widely applied. It has a regular requirement for a system model under consideration and a desired property. Then it systematically checks whether or not the given model satisfies this property. However, the traditional algorithm on Hadoop for checking Computation Tree Logic (CTL) formula does not scale well because of the need of conducting a huge number of states with each passing day. To deal with the state space explosion problem, we propose a scalable algorithm on the platform of Giraph, which is a new rising iterative graph processing platform with high scalability and employs the Bulk Synchronous Parallel (BSP) model. We also address this problem by selecting Probabilistic Computation Tree Logic (PCTL) as the property specification, which is strictly more expressive and makes a great advancement on using probability to describe the probability of an event occurring. This paper comes up with a novel method based on Giraph to verify one PCTL formula, which is used to represent a maximum probability of reaching a certain state, being regarded as a quantitative description of a property in system detection. Owing much to the Giraph platform, our experiment is of great interest as it displays much more efficiently than current methods which are based on Hadoop and so on. Ultimately, we discover that the processing time of Giraph platform applying message model is much shorter than that of Hadoop platform applying iterative model through experimental comparison.

A Distributed Spatiotemporal Contingency Analysis for the Lebanese Power Grid

We address a topological vulnerability analysis of the Lebanese power grid subject to random and cascading failures. Using an Apache Spark implementation that maps the topology of the grid to a complex network, we begin by developing a local structural understanding of the Lebanese power grid that reveals a certain level of decentralization via numerous connected components. Our Apache Spark implementation simulates the random and cascading sequences of events by which energy centers in Lebanon can be exposed and are at risk. The implementation is based on the bulk-synchronous parallel model and maintains optimal work, linear communication time, and a constant number of synchronization barriers. We complement this paper with a spatial understanding of the exposed hotspots. Our results reveal that failures in the power grid are spatially long-range correlated and correlations decay with distance. In a couple of attack scenarios, our Spark implementation achieves significant speedup on 16 cores for a graph with about $9\times 10^{5}$ nodes. Scalability toward 32 nodes improves when experimenting with replicas of the power grid graph which are double and quadruple the original size. This renders this paper suitable to larger networks at many vital levels beyond the power grid.

Model Aggregation Method for Data Parallelism in Distributed Real-Time Machine Learning of Smart Sensing Equipment

In distributed real-time machine learning of smart sensing equipment, training speed and training accuracy are two hard-to-choose trade-off performance measures directly influenced by the design of distributed machine learning algorithms. And it will influence effort of smart sensing equipment directly. We take the model aggregation method of distributed machine learning as a starting point. Due to the loss of accuracy caused by the direct averaging of the parameter average method, we developed the loss function weight reorder stochastic gradient descent method (LR-SGD). LR-SGD uses the loss function value to determine the weight of the work nodes when aggregating the model parameters, and it improves the performance of the parameter average method for nonconvex problems. As shown in the experiment results, our algorithm can improve the training accuracy by a maximum of approximately 0.57% for the Bulk Synchronous Parallel (BSP) model and approximately 6.30% for the Stale Synchronous Parallel (SSP) model.

Полуавтоматический подход к параллельному решению задач с использованием модели Multi-BSP

The Multi-Bulk Synchronous Parallel (Multi-BSP) model is a recently proposed parallel programming model for multicore machines that extends the classic Bulk Synchronous Parallel model. Multi-BSP aims to be a useful model to design algorithms and estimate their running time. This model heavily relies on the right computation of parameters that characterize the hardware. Of course, the hardware utilization also depends on the specific features of the problems and the algorithms applied to solve them. This article introduces a semi-automatic approach for solving problems applying parallel algorithms using the Multi-BSP model. First, the specific multicore computer to use is characterized by applying an automatic procedure. After that, the hardware architecture discovered in the previous step is considered in order to design a portable parallel algorithm. Finally, a fine tuning of parameters is performed to improve the overall efficiency. We propose a specific benchmark for measuring the parameters that characterize the communication and synchronization costs in a particular hardware. Our approach discovers the hierarchical structure of the multicore architecture and compute both parameters for each level that can share data and make synchronizations between computing units. A second contribution of our research is a proposal for a Multi-BSP engine. It allows designing algorithms by applying a recursive methodology over the hierarchical tree already built by the benchmark, focusing on three atomic functions based in a divide-and-conquer strategy. The validation of the proposed method is reported, by studying an algorithm implemented in a prototype of the Multi-BSP engine, testing different parameter configurations that best fit to each problem and using three different high-performance multicore computers.

A communication-reduced and computation-balanced framework for fast graph computation

The bulk synchronous parallel (BSP) model is very user friendly for coding and debugging parallel graph algorithms. However, existing BSP-based distributed graph-processing frameworks, such as Pregel, GPS and Giraph, routinely suffer from high communication costs. These high communication costs mainly stem from the fine-grained message-passing communication model. In order to address this problem, we propose a new computation model with low communication costs, called LCC-BSP. We use this model to design and implement a high-performance distributed graph-processing framework called LCC-Graph. This framework eliminates high communication costs in existing distributed graph-processing frameworks. Moreover, LCC-Graph also balances the computation workloads among all compute nodes by optimizing graph partitioning, significantly reducing the computation time for each superstep. Evaluation of LCC-Graph on a 32-node cluster, driven by real-world graph datasets, shows that it significantly outperforms existing distributed graph-processing frameworks in terms of runtime, particularly when the system is supported by a high-bandwidth network. For example, LCC-Graph achieves an order of magnitude performance improvement over GPS and GraphLab.

Designing and analyzing highly scalable and reliable of full fledged parallel algorithm for computing strongly connected com-ponents to analyze social graphs

With the recent advent of Big Data, developing efficient distributed algorithms for computing Strongly Connected Components of a large dataset has received increasing interests. For example, social networks, information networks and communication networks such as the communities of people that have formed on those networks, what community a person belongs or finding cyclic de-pendencies in the graph.Apache Giraph is an open-source implementation of Google’s Pregel. It is an iterative and real-time graph processing engine designed to be scalable, fault tolerant and highly efficient. This framework provides an accurate platform for the development of parallel algorithms in a distributed environ-ment. It adopts a vertex-centric programming model inspired by Bulk Synchronous Parallel model. A strongly connected component is a maximal sub graph in which all vertices are reachable from every other vertex. Maximal means that it is the largest possible sub graph. It is not possible to find another vertex anywhere in the graph such that it could be added to the sub graph and all the verti-ces in the sub graph would still be connected. In a directed graph G, a pair of vertices u and v are said to be strongly connected to each other if there is a path in each direction between them. Here, we have implemented a parallel algorithm which is based on the new paradigm of graph decomposi-tion for computing strongly connected components. The final outcome mainly focuses on the reduc-tion of total communication costs.

Automatic Cost Analysis for Imperative BSP Programs

Bulk Synchronous Parallel (BSP) is a model for parallel computing with predictable scalability. BSP has a cost model: programs can be assigned a cost which describes their resource usage on any parallel machine. However, the programmer has to manually derive this cost. This paper describes an automatic method for the derivation of BSP program costs, based on classic cost analysis and approximation of polyhedral integer volumes. Our method requires and analyzes programs with textually aligned synchronization and textually aligned, polyhedral communication. We have implemented the analysis and our prototype obtains cost formulas that are parametric in the input parameters of the program and the parameters of the BSP computer and thus bound the cost of running the program with any input on any number of cores. We evaluate the cost formulas and find that they are indeed upper bounds, and tight for data-oblivious programs. Additionally, we evaluate their capacity to predict concrete run times in two parallel settings: a multi-core computer and a cluster. We find that when exact upper bounds can be found, they accurately predict run-times. In networks with full bisection bandwidth, as the BSP model supposes, results are promising with errors < 50%.

An efficient iterative graph data processing framework based on bulk synchronous parallel model

SummaryGraph data processing has been widely applied in a variety of domains such as industry, science, social network, and so on. It therefore has stimulated many efforts devoted to this area. To embrace the fast development trend of big graph data, graph data processing based on Pregel‐like systems has been regarded as one of the most promising ways and has widely attracted the attention of researchers. However, it still remains in its early stage and there still exist many challenges. In Pregel, the superstep synchronization is time consuming as the graph data iteration operation requires multiple synchronizations. Furthermore, the graph data partition strategy adopted by Pregel fails to support load balancing, therefore causing the increase of network I/O overhead as the scale of graph data grows. To address these issues, this paper presents an efficient computational framework for graph data processing based on the bulk synchronous parallel model. The global synchronization control mechanism is improved by determining the start time of the next round of superstep through counting the number of global message files. Furthermore, an improved graph data partition mechanism based on a balanced hash method is proposed to reduce the communication overhead between different partitions of sub‐graph computational tasks. We also re‐design the PageRank algorithm to verify the effectiveness of the proposed framework. Experimental results on different real‐world datasets verify the efficiency of our proposed framework as it outperforms Giraph (an open source Pregel‐like system) by 58%−69%, and achieves 10×−17× performance improvement over Hadoop.

A Lower Bound Technique for Communication in BSP

Communication is a major factor determining the performance of algorithms on current computing systems; it is therefore valuable to provide tight lower bounds on the communication complexity of computations. This article presents a lower bound technique for the communication complexity in the bulk-synchronous parallel (BSP) model of a given class of DAG computations. The derived bound is expressed in terms of the switching potential of a DAG, that is, the number of permutations that the DAG can realize when viewed as a switching network. The proposed technique yields tight lower bounds for the fast Fourier transform (FFT), and for any sorting and permutation network. A stronger bound is also derived for the periodic balanced sorting network, by applying this technique to suitable subnetworks. Finally, we demonstrate that the switching potential captures communication requirements even in computational models different from BSP, such as the I/O model and the LPRAM.