Parallel Framework Research Articles

Edge Intelligence-Assisted Asymmetrical Network Control and Video Decoding in the Industrial IoT with Speculative Parallelization

Industrial Internet of Things (IIoTs) has drawn significant attention in the industry. Among its rich applications, the field’s video surveillance deserves particular interest due to its advantage in better understanding network control. However, existing decoding methods are limited by the video coding order, which cannot be decoded in parallel, resulting in low decoding efficiency and the inability to process the massive amount of video data in real time. In this work, a parallel decoding framework based on the speculative technique is proposed. In particular, the video is first speculatively decomposed into data blocks, and then a verification method is designed to ensure the correctness of the decomposition. After verification, the data blocks having passed the validation can be decoded concurrently in the parallel computing platform. Finally, the concurrent decoding results are concatenated in line with the original encoding order to form the output. Experiments show that compared with traditional serial decoding ones, the proposed method can improve the performance by 9 times on average in the parallel computing environment with NVIDIA Tegra 4 chips, thus significantly enhancing the real-time video data’s decoding efficiency with guaranteed accuracy. Furthermore, proposed and traditional serial methods obtain almost the same peak signal-to-noise ratio (PSNR) and mean square error (MSE) metrics at different bit rates and resolutions, showing that the introduction of the speculative technique does not degrade the decoding accuracy.

Optimized matrix ordering of sparse linear solver using a few-shot model for circuit simulation

The sparse linear solver has become the bottleneck in a SPICE-like circuit simulator. A general sparse linear solver comprises pre-analysis, numeric factorization, and right-hand solving. The matrix ordering method in pre-analysis determines fill-ins and matrix structure, which is critical for the performance of numeric factorization and right-hand solving. Current leading solvers use a sequential (e.g., KLU) or completely parallel (e.g., NICSLU) framework to order the matrix, which produces non-optimal options or consumes tremendous memory. Furthermore, the criterion for judging the optimal ordering method is also controversial. In this paper, we propose a new pre-analysis framework. It redefines the criterion of the optimal ordering method and utilizes a few-shot model for inference. By performing up to 2 ordering methods from the proposed pre-analysis, our work can implement an optimal ordering method and consume much less memory than a completely parallel framework. Experimental results on 70 circuit matrices show that compared to KLU, our work achieves an average time improvement of 27.09%. Compared to NICSLU, our work achieves an average time improvement of 14.39% and an average memory improvement of 61.52% in the sequential mode, and an average time improvement of 15.74% and an average memory improvement of 55.11% in the parallel (16 threads) mode.

Non-uniform Compact Schemes for Parallel Computing

Here, the performance of a sixth order accurate Non-Uniform Compact Scheme (NUCS6) in a parallel framework using the domain decomposition method is analyzed. Traditional compact schemes are used for uniform grids. However, study of flows involving complicated geometries requires transformation of computational domain onto a uniform grid. Compact schemes developed for physical plane computations eliminate the need for such grid transformations. The first successful implementation of uniform compact schemes on a parallelized framework with Schwartz domain decomposition demonstrated the effect of overlap points in eliminating spurious signals at inter-processor boundaries Sengupta et al. [Jour. Comp. Phy. 220 (2007)]. This paper presents the performance of a high order non-uniform compact scheme (NUCS6) in parallel computations with the domain decomposition method. The results of the benchmark problem of flow in a 2D square lid-driven cavity at post-Hopf bifurcation Reynolds number of 10000 is presented to show how the width of overlap region affects the growth and evolution of gyrating polygonal vortex at the centre of the cavity.

Human-centered shared steering control system design for obstacle avoidance scenarios

To minimize the human–machine conflict in the shared steering control system and protect the driving intention of the driver, a human-centered shared steering control system is proposed in obstacle avoidance scenarios. First, a framework is proposed based on the parallel shared steering control system framework, which includes the trajectory reconstruction, driver model, authority allocation model, autonomous controller, and vehicle. And mathematical model of the sub-modules in the shared steering control system is constructed. Then, the trajectory reconstruction method considering the driving intention of the driver is designed. A trigger mechanism with human–machine conflict as input is designed for trajectory reconstruction. More specifically, to protect the driving intention of the driver, the steering input of the driver is used as the constraint to reconstruct the trajectory. Finally, the effectiveness of the proposed method is verified by simulation. The simulation results show that the proposed method can effectively reduce human–machine conflict while the driving intention of the driver is protected. However, the trajectory tracking performance and lateral stability of the vehicle may be may sacrificed within a limited extent.

A Systematic Approach for the Calibration of Route Choice Models Based on Stochastic User Equilibrium

Different from the traditional deterministic user equilibrium (DUE) model based on fixed travel time, the stochastic user equilibrium (SUE) model is established by assuming the perceived route travel time as a random variable. It plays an important role in traffic assignment, wherein the setting of route choice parameters decides system performance. This article studies parameter calibration of widely used SUE models including the multinomial logit (MNL), length-based C-logit (LCL), and congestion-based C-logit (CCL) models. We establish a bilevel model in which the upper level is designed to minimize the error between the real and model-based traffic flow patterns, whereas the lower level copes with the SUE traffic assignment. A series of innovative methods are built to solve this bilevel model. Considering the nonlinear calibration problem in the upper level, a Bayesian optimization method is used. For the SUE traffic assignment in the lower level, dedicated algorithms are proposed, which combine a partial linearization algorithm and a feasible direction method. Furthermore, two revised solution methods are provided: a Barzilai–Borwein (BB) step-size method and a parallel framework. Finally, the proposed model and corresponding solution approach are validated based on a case study of Yuyao, China. Solutions show that the CCL SUE model has the highest precision while requiring the longest computation time. LCL takes only 11.7% of the computation time of CCL when reaching an acceptable precision.

The formation of specific and gist associative episodic memory representations during encoding: Effects of rate of presentation.

Assessing the time course under which underlying memory representations can be formed is an important question for understanding memory. Several studies assessing item memory have shown that gist representations of items are laid out more rapidly than verbatim representations. However, for associations among items/components, which form the core of episodic memory, it is unclear whether gist representations form more quickly than, or at least in parallel with, verbatim representations, as fuzzy-trace theory predicts, or whether gist is extracted more slowly from inferring the meaning of verbatim representations, as in gist macroprocessor theories. To test these contrasting possibilities, we used a novel associative recognition task in which participants studied face-scene pairs for .75, 1.5, or 4 seconds each, and were later tested on their ability to discriminate intact pairs from foils which varied in how similar they were to originally studied pairs. Across 2 experiments, we found that verbatim memory for associations, measured using a multinomial-processing-tree model, improved from .75 to 1.5 to 4 seconds of presentation time. Paralleling these effects of encoding time on verbatim memory, for gist memory, there were improvements from .75 seconds to 1.5 seconds in both experiment 1 and 2, while improvements from 1.5 seconds to 4 seconds were only evident when the retention interval between study and test was increased (experiment 2). These results provide strong support for the parallel processing framework of fuzzy-trace theory over the slow gist extraction framework of an alternative gist macroprocessor theory. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Automatic Pipeline Parallelism: A Parallel Inference Framework for Deep Learning Applications in 6G Mobile Communication Systems

With the rapid development of wireless communication, achieving the neXt generation Ultra-Reliable and Low-Latency Communications (xURLLC) in 6G mobile communication systems has become a critical problem. Among many applications in xURLLC, deep learning model inference requires improvement over its efficiency. Due to the heterogeneous hardware environment in 6G, parallel schedules from distributed machine learning and edge computing has been borrowed to tackle the efficiency problem. However, traditional parallel schedules suffer from high latency, low throughput, and low device utility. In this paper, we propose Automatic Pipeline Parallelism ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">AP</i> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> ), a parallel inference framework for deep learning applications in 6G mobile communication systems, to improve the model inference efficiency while maintaining reliability. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">AP</i> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> contains three sub-modules. A task-device affinity predictor predicts a task’s expected execution time on a given device. The parallel inference arrangement optimizer finds the most suitable device for each task. The parallel inference scheduler converts the arrangement to a schedule that can be directly executed in the system. The experimental results show that <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">AP</i> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> can achieve better latency, throughput, reliability, and device utility than other parallel schedules. Also, the priority of the sub-module designs has been approved through the experiments.

AutoGraph: Optimizing DNN Computation Graph for Parallel GPU Kernel Execution

Deep learning frameworks optimize the computation graphs and intra-operator computations to boost the inference performance on GPUs, while inter-operator parallelism is usually ignored. In this paper, a unified framework, AutoGraph, is proposed to obtain highly optimized computation graphs in favor of parallel executions of GPU kernels. A novel dynamic programming algorithm, combined with backtracking search, is adopted to explore the optimal graph optimization solution, with the fast performance estimation from the mixed critical path cost. Accurate runtime information based on GPU Multi-Stream launched with CUDA Graph is utilized to determine the convergence of the optimization. Experimental results demonstrate that our method achieves up to 3.47x speedup over existing graph optimization methods. Moreover, AutoGraph outperforms state-of-the-art parallel kernel launch frameworks by up to 1.26x.

A parallel particle swarm optimization framework based on a fork-join thread pool using a work-stealing mechanism

Particle Swarm Optimization (PSO) is one of the most popular optimization algorithms that has been adopted in various fields, including design, scheduling, and biochemistry. However, the algorithm is time-consuming when facing high-dimensional or multi-objective optimizing problems. Parallel PSO is thus proposed to improve its computing efficiency, and many studies have been conducted. However, the low-level system design is seldom considered in parallel programming, which may have a nonnegligible impact on computing efficiency, creating a gap between low-level optimization and high-level algorithm design. Therefore, this paper proposes a Parallel Asynchronous PSO (PAPSO) framework based on thread pools utilizing multicore processors and adopts a cross-level approach to bridge the gap. A series of experiments are designed and conducted to examine how the aforementioned method can improve the parallel execution efficiency of PSO compared with the OpenMP framework and nonparallel PSO. Results indicate that PAPSO can significantly improve PSO computing efficiency by reducing the elapsed time, approaching approximately 20% optimization compared with OpenMP. Additionally, it achieves an approximately linear speedup of up to 4.5 threads. In addition, the particle communication experiment shows that the nonblocking communication protocol is effective for maintaining the computing elapsed time in the same level facing different neighborhood sizes. Finally, the work-stealing mechanism achieves an average of 16% improvement for general computing scenarios and maintain up to 16% improvement for imbalanced workload scenarios. Generally, the major contribution of this paper is that we innovate the thread pooling management concept with the fork-join model in parallelizing PSO to make sufficient use of multiple core CPUs for parallel computing through multithread programming.

Parallel Peeling of Bipartite Networks for Hierarchical Dense Subgraph Discovery

Wing and Tip decomposition are motif-based analytics for bipartite graphs that construct a hierarchy of butterfly (2,2-biclique) dense edge and vertex induced subgraphs, respectively. They have applications in several domains, including e-commerce, recommendation systems, document analysis, and others. Existing decomposition algorithms use a bottom-up approach that constructs the hierarchy in an increasing order of the subgraph density. They iteratively select the edges or vertices with minimum butterfly count peel , i.e., remove them along with their butterflies. The amount of butterflies in real-world bipartite graphs makes bottom-up peeling computationally demanding. Furthermore, the strict order of peeling entities results in a large number of sequentially dependent iterations. Consequently, parallel algorithms based on bottom up peeling incur heavy synchronization and poor scalability. In this article, we propose a novel Parallel Bipartite Network peelinG (PBNG) framework that adopts a two-phased peeling approach to relax the order of peeling, and in turn, dramatically reduce synchronization. The first phase divides the decomposition hierarchy into few partitions and requires little synchronization. The second phase concurrently processes all partitions to generate individual levels of the hierarchy and requires no global synchronization. The two-phased peeling further enables batching optimizations that dramatically improve the computational efficiency of PBNG. We empirically evaluate PBNG using several real-world bipartite graphs and demonstrate radical improvements over the existing approaches. On a shared-memory 36 core server, PBNG achieves up to 19.7× self-relative parallel speedup. Compared to the state-of-the-art parallel framework P ar B utterfly , PBNG reduces synchronization by up to 15,260× and execution time by up to 295×. Furthermore, it achieves up to 38.5× speedup over state-of-the-art algorithms specifically tuned for wing decomposition. Our source code is made available at https://github.com/kartiklakhotia/RECEIPT .

CarNum: parallel numerical framework for computational cardiac electromechanics

Abstract A new parallel numerical framework CarNum is presented for efficient coupling of mathematical models in multiphysics problems such as computational cardiac electromechanics. This framework is based on open source projects, which provide the core functionality of the platform. Computational cardiac electromechanics requires a complex pipeline of solving different types of ordinary and partial differential equations. Our framework allows one to implement different numerical schemes and provides more control in multiphysics coupling. This paper outlines a concept of the new platform and details of numerical modelling of cardiac electromechanics. First experiments with well-known cardiac electromechanics benchmarks show good agreement with other groups and decent parallel scalability.

PaPILO: A Parallel Presolving Library for Integer and Linear Optimization with Multiprecision Support

Presolving has become an essential component of modern mixed integer program (MIP) solvers, both in terms of computational performance and numerical robustness. In this paper, we present PaPILO, a new C++ header-only library that provides a large set of presolving routines for MIP and linear programming problems from the literature. The creation of PaPILO was motivated by the current lack of (a) solver-independent implementations that (b) exploit parallel hardware and (c) support multiprecision arithmetic. Traditionally, presolving is designed to be fast. Whenever necessary, its low computational overhead is usually achieved by strict working limits. PaPILO’s parallelization framework aims at reducing the computational overhead also when presolving is executed more aggressively or is applied to large-scale problems. To rule out conflicts between parallel presolve reductions, PaPILO uses a transaction-based design. This helps to avoid both the memory-intensive allocation of multiple copies of the problem and special synchronization between presolvers. Additionally, the use of Intel’s Threading Building Blocks library aids PaPILO in efficiently exploiting recursive parallelism within expensive presolving routines, such as probing, dominated columns, or constraint sparsification. We provide an overview of PaPILO’s capabilities and insights into important design choices. History: Accepted by Ted Ralphs, Area Editor for Software Tools. Funding: This work has been financially supported by Research Campus MODAL, funded by the German Federal Ministry of Education and Research [Grants 05M14ZAM, 05M20ZBM], and the European Union’s Horizon 2020 research and innovation programme under grant agreement No 773897 (plan4res). The content of this paper only reflects the author’s views. The European Commission / Innovation and Networks Executive Agency is not responsible for any use that may be made of the information it contains. Supplemental Material: The software that supports the findings of this study is available within the paper and its Supplemental Information ( https://pubsonline.informs.org/doi/suppl/10.1287/ijoc.2022.0171 ), as well as from the IJOC GitHub software repository ( https://github.com/INFORMSJoC/2022.0171 ). The complete IJOC Software and Data Repository is available at https://informsjoc.github.io/ .

Optimizing DNNs Model Partitioning for Enhanced Performance on Edge Devices

Deep Neural Networks (DNNs) have proven effective in various applications due to their dominant performance. However, integrating DNNs into edge devices remains challenging due to the large size of the DNN model, which requires efficient model parallelization and workload partitioning. Previous attempts to address these challenges have focused on data and model parallelism but have fallen short in terms of finding the optimal DNN model partitions for efficient distribution, considering available resources. This paper presents a pipelined DNN model parallelism framework that improves the performance of DNNs on edge devices. The framework optimizes DNN model training by determining the optimal number of partitions based on available edge resources. This is achieved through a combination of data and model parallelism techniques, which efficiently distribute the workload across multiple processors to reduce training time. The framework also includes a task controller to manage computing resources effectively. The experimental results demonstrate the effectiveness of the proposed approach, showing a significant reduction in the model training time compared to a baseline model AlexNet.

A Parallel Framework for Constraint-Based Bayesian Network Learning via Markov Blanket Discovery

Bayesian networks (BNs) are a widely used graphical model in machine learning. As learning the structure of BNs is NP-hard, high-performance computing methods are necessary for constructing large-scale networks. In this paper, we present a parallel framework to scale BN structure learning algorithms to tens of thousands of variables. Our framework is applicable to learning algorithms that rely on the discovery of Markov blankets (MBs) as an intermediate step. We demonstrate the applicability of our framework by parallelizing three different algorithms: <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Grow-Shrink</i> ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">GS</i> ), <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Incremental Association MB</i> ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">IAMB</i> ), and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Interleaved IAMB</i> ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Inter-IAMB</i> ). Our implementations are available as part of an open-source software called <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ramBLe</i> , and are able to construct BNs from real data sets with tens of thousands of variables and thousands of observations in less than a minute on 1024 cores, with a speedup of up to 845X and 82.5% efficiency. Furthermore, we demonstrate using simulated data sets that our proposed parallel framework can scale to BNs of even higher dimensionality. Our implementations were selected for the reproducibility challenge component of the 2021 student cluster competition (SCC'21), which tasked undergraduate teams from around the world with reproducing the results that we obtained using the implementations. We discuss details of the challenge and the results of the experiments conducted by the top teams in the competition. The results of these experiments indicate that our key results are reproducible, despite the use of completely different data sets and experiment infrastructure, and validate the scalability of our implementations.

Critique of “A Parallel Framework for Constraint-Based Bayesian Network Learning via Markov Blanket Discovery” by SCC Team From Tsinghua University

Srivastava et al. propose a parallel framework to optimize Bayesian network learning in the SC20 paper entitled “A Parallel Framework for Constraint-Based Bayesian Network Learning via Markov Blanket Discovery”. They parallelize all the phases in network constructing algorithms to achieve high performance and scalability. In this paper, we reproduce the strong scaling and weak scaling experiments in that SC paper. We conduct experiments on a 4-node cluster with Intel CPUs provided by the SCC committee. We further analyze the results of communication overhead. Our results show that the proposed method in that SC paper scales well on the provided cluster, in accordance with the SC paper.

Critique of “A Parallel Framework for Constraint-Based Bayesian Network Learning via Markov Blanket Discovery” by SCC Team From ShanghaiTech University

In SC20, Srivastava <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">et al.</i> [6] proposed a Parallel F <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ram</b> ework for <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">B</b> ayesian <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Le</b> arning, or ramBLe, for short, which is a highly parallel and efficient framework for learning the structure of Bayesian Networks (BNs) from samples, particularly large genome-scale networks. As part of our participation in the SC21 Student Cluster Competition, our task was to verify conclusions from the original work [6]. Here we present the outcome of our experiments, which were performed on a four-node cluster from the Oracle Cloud HPC platform. We reproduce the numerical results from [6], namely the algorithm's performance and scaling behavior using MPI and different Python and Boost libraries on the Oracle cloud.

The Future of Management: DAO to Smart Organizations and Intelligent Operations

In the future, management in smart societies will revolve around knowledge workers and the works they produce. This article is committed to explore new management framework, model, paradigm, and solution for organizing, managing, and measuring knowledge works. First, the parallel management framework is presented that would allow for the virtual-real interactions of humans in social space, robots in physical space, and digital humans in cyberspace to realize descriptive, predictive, and prescriptive intelligence for management. Then, the management foundation models are proposed by fusing scenarios engineering with artificial intelligence foundation models and cyber–physical-social systems. Moreover, the new management paradigm driven by decentralized autonomous organizations and operations is formulated for the advancement of smart organizations and intelligent operations. On these basis, the management operating systems that highlight features of simple intelligence, provable security, flexible scalability, and ecological harmony are finally put forward as new management solution.

Adaptive Multidimensional Parallel Fault Simulation Framework on Heterogeneous System

Adaptive Multidimensional Parallel Fault Simulation Framework on Heterogeneous System

Critique of: “A Parallel Framework for Constraint-Based Bayesian Network Learning via Markov Blanket Discovery” by SCC Team From UC San Diego

Bayesian networks (BNs) have become popular in recent years to describe natural phenomena in situations where causal linkages are important to understand. In order to get around the inherent non-tractability of learning BNs, Srivastava et al. propose a markov blanket discovery-based approach to learning in their paper titled <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">”A Parallel Framework for Constraint-based Bayesian Network Learning via Markov Blanket Discovery.”</i> We are able to reproduce both the strong and weak scaling experiments from the paper up to 128 cores, and verify communication cost scaling for all three algorithms in the paper. We also introduce methodological improvements to weak scaling that show the paper's findings are unique to the methodology and not the datasets used. Slight variations in performance were observed due to differences in datasets, core count, and job scheduling.

Critique of “A Parallel Framework for Constraint-Based Bayesian Network Learning via Markov Blanket Discovery” by SCC Team From Peking University

Ankit Srivastava et al. [1] proposed a parallel framework for Constraint-Based Bayesian Network (BN) Learning via Markov Blanket Discovery (referred to as ramBLe) and implemented it over three existing BN learning algorithms, namely, GS, IAMB and Inter-IAMB. As part of the Student Cluster Competition at SC21, we reproduce the computational efficiency of ramBLe on our assigned Oracle cluster. The cluster has 4x36 cores in total with 100 Gbps RoCE v2 support and is equipped with Centos-compatible Oracle Linux. Our experiments, covering the same three algorithms of ramBLe, evaluate its strong and weak scalability of the algorithms using real COVID-19 data sets. We verify part of the conclusions in the paper and propose our explanation of the differences.