Synchronous Parallel Research Articles

Self-Powered Synchronized Switching Interface Circuit for Piezoelectric Footstep Energy Harvesting.

Piezoelectric Vibration converters are nowadays gaining importance for supplying low-powered sensor nodes and wearable electronic devices. Energy management interfaces are thereby needed to ensure voltage compatibility between the harvester element and the electric load. To improve power extraction ability, resonant interfaces such as Parallel Synchronized Switch Harvesting on Inductor (P-SSHI) have been proposed. The main challenges for designing this type of energy management circuits are to realise self-powered solutions and increase the energy efficiency and adaptability of the interface for low-power operation modes corresponding to low frequencies and irregular vibration mechanical energy sources. In this work, a novel Self-Powered (SP P-SSHI) energy management circuit is proposed which is able to harvest energy from piezoelectric converters at low frequencies and irregular chock like footstep input excitations. It has a good power extraction ability and is adaptable for different storage capacitors and loads. As a proof of concept, a piezoelectric shoe insole with six integrated parallel piezoelectric sensors (PEts) was designed and implemented to validate the performance of the energy management interface circuit. Under a vibration excitation of 1 Hz corresponding to a (moderate walking speed), the maximum reached efficiency and power of the proposed interface is 83.02% and 3.6 mW respectively for the designed insole, a 10 kΩ resistive load and a 10 μF storage capacitor. The enhanced SP-PSSHI circuit was validated to charge a 10 μF capacitor to 6 V in 3.94 s and a 1 mF capacitor to 3.2 V in 27.64 s. The proposed energy management interface has a cold start-up ability and was also validated to charge a (65 mAh, 3.1 V) maganese dioxide coin cell Lithium battery (ML 2032), demonstrating the ability of the proposed wearable piezoelectric energy harvesting system to provide an autonomous power supply for wearable wireless sensors.

FSP: Towards Flexible Synchronous Parallel Frameworks for Distributed Machine Learning

Myriad of machine learning (ML) algorithms refine model parameters iteratively. Existing synchronous data-parallel frameworks can accelerate training with convergence guarantees. However, the pre-assigned workload-based synchronous design still poses great challenges, since fast workers must wait for slow, straggling ones, especially in a heterogeneous computing cluster. Asynchronous alternatives can bypass this performance bottleneck, but at expense of potentially losing convergence guarantees. This article proposes a new time-based flexible synchronous parallel framework (FSP). It provides strict convergence analysis by consistently updating parameters, as well as significant cost reduction by completely unleashing the power of fast workers. It identifies the optimal synchronization frequency, by online balancing costs of parameters’ update and benefits brought by their freshness. Besides the basic goal of keeping all workers fully CPU-utilized, FSP also aims to keep data spread over the cluster fully utilized, so that they can contribute to convergence with equal opportunities. These proposals are all implemented in a prototype system Flegel, with additional engineering optimizations for further performance enhancement and programming facilitation. Experiments demonstrate that Flegel significantly outperforms recent studies.

The convergence of pharmacometrics and quantitative systems pharmacology in pharmaceutical research and development

Quantitative systems pharmacology (QSP) models are an important facet of pharmaceutical and clinical research as they combine mechanistic models of physiology in health and disease with pharmacokinetics/pharmacodynamics to predict systems-level effects. The quantitative clinical pharmacology toolbox has traditionally included both mechanistic modeling and population approaches, collectively called pharmacometrics, but the current landscape requires the optimization and use of multiple models together. Here, we explore several case studies in drug development that exemplify three approaches for using QSP and pharmacometrics models together - parallel synchronization, cross-informative use, and sequential integration. While these approaches are increasingly applied in drug development, achieving a true convergence of QSP and pharmacometrics that fully exploits their synergy will require new tools and methods that enable greater technical integration, in addition to nurturing scientists with diverse modeling expertise that enable cross-discipline strategy. Extensions of existing methods used in each approach as well as additional resources including machine learning models, data-at-scale, end-to-end computation platforms, and real-time analytics will enable this convergence.

Comprehensive Mutation Profiling of Colorectal Cancer Patients With Lung or Liver Metastasis by Targeted Next-Generation Sequencing.

Primary tumor tissue is often analyzed to search for predictive biomarkers and DNA-guided personalized therapies, but there is an incomplete understanding of the discrepancies in the genomic profiles between primary tumors and metastases, such as liver and lung metastases. We performed in-depth targeted next-generation sequencing of 520 key cancer-associated genes for 47 matched primary and metastatic tumor samples which were retrospectively collected. A total of 699 mutations were detected in the 47 samples. The coincidence rate of primary tumors and metastases was 51.8% (n = 362), and compared to patients with liver metastases, patients with lung metastases had a significantly greater coincidence rate (P = .021). The number of specific mutations for the primary tumors and liver and lung metastases was 186 (26.6%), 122 (17.5%), and 29 (4.1%), respectively. Analysis of a patient with all three occurrences, including a primary tumor, liver metastasis, and lung metastasis, indicated a possible polyclonal seeding mechanism for liver metastases. Remarkably, multiple samples from patients with primary and metastatic tumors supported a mechanism of synchronous parallel dissemination from primary tumors to metastatic tumors that were not mediated through pre-metastatic tumors. We also found that the PI3K-Akt signaling pathway significantly altered lung metastases compared to matched primary tumors (P = .001). In addition, patients with mutations in CTCF, PIK3CA, or TP53 and LRP1B, AURKA, FGFR1, ATRX, DNMT3B, or GNAS had larger primary tumor sizes and metastases, especially patients with both LRP1B and AURKA mutations. Interestingly, CRC patients with TP53-disruptive mutations were more likely to have liver metastases (P = .016). In this study, we demonstrate significant differences in the genomic landscapes of colorectal cancer patients based on the site of metastasis. Notably, we observe a larger genomic variation between primary tumors and liver metastasis compared to primary tumors and lung metastasis. These findings can be used for tailoring treatments based on the specific metastatic site.

On the convergence analysis of asynchronous SGD for solving consistent linear systems

Parallel and distributed stochastic algorithms have drawn significant attention in the realm of big data and machine learning. While the synchronous versions of these algorithms are well understood in terms of their convergence, the convergence analyses of their asynchronous counterparts are not widely studied. In this paper, we propose and analyze a distributed, asynchronous parallel algorithm to solve an arbitrary, consistent linear system by reformulating the system into a stochastic optimization problem as Richtárik and Takác̃ in [1]. We compare the convergence rates of our asynchronous algorithm with the synchronous parallel algorithm proposed by Richtárik and Takáč in [1] under different choices of the hyperparameters—the stepsize, the damping factor, the number of processors, and the delay factor. We show that our asynchronous parallel algorithm enjoys a global linear convergence rate, similar to the synchronous parallel algorithm in [1] under the same setup. We also show that our asynchronous algorithm improves upon the synchronous algorithm in [1] with a better convergence rate when the number of processors is larger than four. Furthermore, our asynchronous parallel algorithm performs asymptotically better than its synchronous counterpart for certain linear systems. Finally, we compute the minimum number of processors required for those systems for which our asynchronous algorithm is better and find that this number can be as low as two for some ill-conditioned problems.

Adaptive synchronous strategy for distributed machine learning

In distributed machine learning training, bulk synchronous parallel (BSP) and asynchronous parallel (ASP) are two main synchronization methods to help achieve gradient aggregation. However, BSP needs longer training time due to “stragglers” problem, while ASP sacrifices the accuracy due to “gradient staleness” problem. In this article, we propose a distributed training paradigm on parameter server framework called adaptive synchronous strategy (A2S) which improves the BSP and ASP paradigms by adaptively adopting different parallel training schemes for workers with different training speeds. Based on the stale value between the fastest and slowest worker, A2S adaptively adds a relaxed synchronous barrier for fast workers to alleviate gradient staleness, where a differentiated weighting gradient aggregation method is used to reduce the impact of slow gradients. Simultaneously, A2S adopts ASP training for slow workers to eliminate stragglers. Hence, A2S not only improves the “gradient staleness” and “stragglers” problems, but also obtains convergence stability and synchronous gain through synchronous and asynchronous parallel, respectively. Specially, we theoretically proved the convergence of A2S by deriving the regret bound. Moreover, experiment results show that A2S improves accuracy by up to 2.64% and accelerates training by up to 41% more than the state-of-the-art synchronization methods BSP, ASP, stale synchronous parallel (SSP), dynamic SSP, and Sync-switch.

A Generalized Framework of Multifidelity Max-Value Entropy Search Through Joint Entropy.

Bayesian optimization (BO) is a popular method for expensive black-box optimization problems; however, querying the objective function at every iteration can be a bottleneck that hinders efficient search capabilities. In this regard, multifidelity Bayesian optimization (MFBO) aims to accelerate BO by incorporating lower-fidelity observations available with a lower sampling cost. In our previous work, we proposed an information-theoretic approach to MFBO, referred to as multifidelity max-value entropy search (MF-MES), which inherits practical effectiveness and computational simplicity of the well-known max-value entropy search (MES) for the single-fidelity BO. However, the applicability of MF-MES is still limited to the case that a single observation is sequentially obtained. In this letter, we generalize MF-MES so that information gain can be evaluated even when multiple observations are simultaneously obtained. This generalization enables MF-MES to address two practical problem settings: synchronous parallelization and trace-aware querying. We show that the acquisition functions for these extensions inherit the simplicity of MF-MES without introducing additional assumptions. We also provide computational techniques for entropy evaluation and posterior sampling in the acquisition functions, which can be commonly used for all variants of MF-MES. The effectiveness of MF-MES is demonstrated using benchmark functions and real-world applications such as materials science data and hyperparameter tuning of machine-learning algorithms.

High Gain, Load-Tolerant Self-Powered Series–Parallel Synchronized Switching Technique for Piezoelectric Energy Harvesting

In the framework of autonomous wireless devices ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g.</i> , sensors) powered by energy available from their surrounding environment, this article exposes a technique combining the so-called series and parallel Synchronized Switch Harvesting on Inductor (SSHI) approaches for piezoelectric energy harvesting. More specifically, the proposed circuit consists in switching the piezoelectric element on two inductors, one in series between the piezoelement and the energy extraction stage (rectifier bridge) and the other one in parallel with the rectifier. This, therefore, allows several energy transfer processes to occur per half period, even when the switching process is finished. Based on numerical analysis supported by experimental validations, it is shown that such a technique permits a significant widening of the effective load and rectified voltage ranges, but also, under particular conditions, a power output magnification that is even larger than in the sole SSHI case.

Application of electromagnetic continuous variable transmission in hydraulic turbines to increase stability of an off-grid power system

In the context of climate change, today's adoption of renewable energy sources in isolated power systems is becoming an increasingly more significant issue. Micro-hydro power plants are emerging as a mature balancing technology and a great alternative to large hydropower plants as they do not encounter population displacement and many environmental problems. It is worth noting that hydroelectric power plants combined with various sources require improvement in control laws and algorithms. Strong disturbing impacts, namely transient processes, in an isolated electric power system cause an appreciable decrease in dynamic stability. This paper presents research conducted to ensure synchronous parallel operation of permanent-magnet generators used in micro-hydroelectric power plants in an isolated power system with a controllable flexible coupling between turbine and generator. The importance of this study lies in the issue of electromechanical compatibility to be achieved in parallel operation of generators that have different parameters of time constant of mechanical inertia of rotors of electrical machines. This paper provides an overview of possible solutions to the identified problem. A concept based on a controllable flexible coupling between generator and turbine is proposed as a promising avenue to solve this problem. The mathematical models are presented along with the laws and algorithms developed for automatic control of the generator speed. Oscillograms of simulation and physical modeling presented in the final part of the paper show parallel operation of permanent-magnet synchronous generators in an isolated electric power system for various electrical distances of three-phase short circuit points.

The research of begin the Apache Giraph

Apache Giraph is an open-source image processing framework built from Apache Hadoop. Apache Giraph is based on bulk synchronous parallel and Google’s Pregel. It is suitable for running large-scale logical calculations, such as page ranking, shared links, personalized ranking, etc. Giraph focuses on social graph computing and is at the core of Facebook’s open graph tool, handling trillions of connections between users and their behaviors in a few minutes. But because the time is limited, the work just has been done with Hadoop. This paper will be mainly talked about Hadoop and how the difficulty to be solved.

Dynamic Allocation Method of Economic Information Integrated Data Based on Deep Learning Algorithm.

In view of the low efficiency of traditional data fusion algorithms in wireless sensor networks and the difficulty in processing high-dimensional data, a new algorithm CNNMDA, based on the deep learning model is proposed to realize data fusion. Firstly, the algorithm trains the constructed feature extraction model CNNM at the sink node; then each terminal node extracts the original data features through CNNM and finally sends the fused data to the sink node, so as to reduce the data transmission amount and prolong the network life. Simulation experiments show that compared with similar fusion algorithms, the CNNMDA can greatly reduce network energy consumption under the same data amount, and effectively improve the efficiency and accuracy of data fusion. In order to solve the problem that parameter synchronization takes too long in synchronous parallel, a dynamic training data allocation algorithm in multimachine synchronous parallel is proposed. Based on the computing efficiency of compute nodes, the amount of sample data to be processed by nodes will be dynamically adjusted after each iteration. This mechanism not only enables the model to be synchronized and parallel but also reduces the time of waiting for gradient updates. Finally, a comparative experiment is carried out on the Tianhe-2 supercomputer, and the experimental results show that the proposed optimization mechanism achieves the expected effect.

Correction to: FLSGD: free local SGD with parallel synchronization

Correction to: FLSGD: free local SGD with parallel synchronization

Federated Learning With Heterogeneity-Aware Probabilistic Synchronous Parallel on Edge

With the massive amount of data generated from mobile devices and the increase of computing power of edge devices, the paradigm of Federated Learning has attracted great momentum. In federated learning, distributed and heterogeneous nodes collaborate to learn model parameters. However, while providing benefits such as privacy by design and reduced latency, the heterogeneous network present challenges to the synchronisation methods, or barrier control methods, used in training, regarding system progress and model convergence etc. The design of these barrier mechanisms is critical for the performance and scalability of federated learning systems. We propose a new barrier control technique called Probabilistic Synchronous Parallel (PSP). In contrast to existing mechanisms, it introduces a sampling primitive that composes with existing barrier control mechanisms to produce a family of mechanisms with improved convergence speed and scalability. Our proposal is supported with a convergence analysis of PSP-based SGD algorithm. In practice, we also propose heuristic techniques that further improve the efficiency of PSP. We evaluate the performance of proposed methods using the federated learning specific FEMNSIT dataset. The evaluation results show that PSP can effectively achieve good balance between system efficiency and model accuracy, mitigating the challenge of heterogeneity in federated learning.

An efficient and robust parallel scheduler for bioinformatics applications in a public cloud: A bigdata approach

&lt;p&gt;In bioinformatics, genomic sequence alignment is a simple method for handling and analysing data, and it is one of the most important applications in determining the structure and function of protein sequences and nucleic acids. The basic local alignment search tool (BLAST) algorithm, which is one of the most frequently used local sequence alignment algorithms, is covered in detail here. Currently, the NCBI's BLAST algorithm (standalone) is unable to handle biological data in the terabytes. To address this problem, a variety of schedulers have been proposed. Existing sequencing approaches are based on the Hadoop MapReduce (MR) framework, which enables a diverse set of applications and employs a serial execution strategy that takes a long time and consumes a lot of computing resources. The author, improves the BLAST algorithm based on the BLAST-BSPMR algorithm to achieve the BLAST algorithm. To address the issue with Hadoop's MapReduce framework, a customised MapReduce framework is developed on the Azure cloud platform. The experiment findings indicate that the suggested bulk synchronous parallel MapReduce-basic local alignment search tool (BSPMR-BLAST) algorithm matches bioinformatics genomic sequences more quickly than the existing Hadoop-BLAST method, and that the proposed customised scheduler is extremely stable and scalable.&lt;/p&gt;

A Comprehensive Inspection of the Straggler Problem

Parameter server is a popular distributed processing paradigm for operating distributed deep learning (DL) applications. As a growing number of DL models are trained via shared clusters, machines are in confrontation with the heterogeneous environment, which incurs the unexpected phenomenon with a slow task processing speed called straggler. Straggler addressing is a crucial issue in distributed DL applications, since stragglers significantly hamper system performance. While many techniques have been deployed to mitigate stragglers, they may not achieve their goals with the presence of heterogeneity, where systems consume much longer time until DL training convergence than in a homogeneous environment, as evidenced by our experimental study. With the methodology of straggler projection and abstraction of parallelism, a new synchronization mechanism called elastic parallelism synchronous parallel (EPSP) is proposed, which exploits the superiority of iteration acceleration in stale synchronous parallel and conquers the shortage of barrier wasting time in bulk synchronous parallel. More precisely, EPSP supports both enforced and slack synchronization by adjusting the parameter of staleness.

ZipLine: an optimized algorithm for the elastic bulk synchronous parallel model

The bulk synchronous parallel (BSP) is a celebrated synchronization model for general-purpose parallel computing that has successfully been employed for distributed training of deep learning models. A shortcoming of the BSP is that it requires workers to wait for the straggler at every iteration. Therefore, employing BSP increases the waiting time of the faster workers of a cluster and results in an overall prolonged training time. To ameliorate this shortcoming of BSP, we propose ElasticBSP, a model that aims to relax its strict synchronization requirement with an elastic synchronization by allowing delayed synchronization to minimize the waiting time. ElasticBSP offers more flexibility and adaptability during the training phase, without sacrificing the accuracy of the trained model. ElasticBSP is realized by the algorithm named ZipLine, which consists of two phases. First, it estimates for each worker the end time points of its future iterations at run time, and then a one-pass algorithm over the estimated time points of all workers is employed to fast compute an optimal future time point for synchronization. We provide theoretical results about the correctness and performance of the ZipLine algorithm. Furthermore, we propose algorithmic and implementation optimizations of ZipLine, namely ZipLineOpt and ZipLineOptBS, which reduce the time complexity of ZipLine to linearithmic time. A thorough experimental evaluation demonstrates that our proposed ElasticBSP model, materialized by the proposed optimized ZipLine variants, converges faster and to a higher accuracy than the predominant BSP. The focus of the paper is on optimizing the synchronization scheduling over a parameter server architecture. It is orthogonal to other types of optimizations, such as the learning rate optimization.

EXPPO: EXecution Performance Profiling and Optimization for CPS Co-simulation-as-a-Service

A co-simulation may comprise several heterogeneous federates with diverse spatial and temporal execution characteristics. In an iterative time-stepped simulation, a federation exhibits the Bulk Synchronous Parallel (BSP) computation paradigm in which all federates perform local operations and synchronize with their peers before proceeding to the next round of computation. In this context, the lowest performing (i.e., slowest) federate dictates the progression of the federation logical time. One challenge in co-simulation is performance profiling for individual federates and entire federations. The computational resource assignment to the federates can have a large impact on federation performance. Furthermore, a federation may comprise federates located on different physical machines as is the case for cloud and edge computing environments. As such, distributed profiling and resource assignment to the federation is a major challenge for operationalizing the co-simulation execution at scale. This paper presents the Execution Performance Profiling and Optimization (EXPPO) methodology, which addresses these challenges by using execution performance profiling at each simulation execution step and for every federate in a federation. EXPPO uses profiling to learn performance models for each federate, and uses these models in its federation resource recommendation tool to solve an optimization problem that improves the execution performance of the co-simulation. Using an experimental testbed, the efficacy of EXPPO is validated to show the benefits of performance profiling and resource assignment in improving the execution runtimes of co-simulations while also minimizing the execution cost.

Parallel Business Processes Modeling

The article considers a new section of the modern theory of business processes dedicated to their parallel processing. The relevance of the work is due to the need to reorganize business processes in order to improve a number of their key indicators, such as speed, cost, etc. It should be noted that the well-known models and methods of the modern theory of business processes practically do not address the issues of parallelism in them, although this direction occupies a significant place in the general theory of processes and, in addition to the actual methods of description, analysis and verification of parallel structures, also includes methods of parallelization of sequential structures. The paper proposes a classification of parallel business processes based on the types of parallel computing systems and completely covering the subject area under consideration. Examples of parallel business processes of each type are given. For each of the selected classes of business processes, a brief overview of the current state of affairs in terms of parallelism is given; the directions of relevant work are identified and analyzed. Brief descriptions of models and methods of pipelined and synchronous parallelism are given. In terms of asynchronous parallelism, a model and language tools for organizing parallel independent branches (processes) are proposed. A mixed graph containing nodes and branches of various types is used as the corresponding model. The scenario of executing a parallel business process is formalized. To estimate the execution time of a parallel business process, the basic theorems of the theory of computational processes are adapted in order to focus them on business processes, recommendations are given on the rules for building parallel business processes. The requirements for the language tools for describing asynchronous parallel business processes are formulated. Next, the methods of parallelization of business processes related to its linear sections and cycles are considered. The definitions of the linear section and the cycle in the business process are introduced. For their parallelization, the key task is to identify information dependencies between the functions of the business process. Attention is drawn to the sequence of definitions/uses of information objects, manifested through incoming and outgoing information flows for functional objects of the corresponding visual modeling language. In terms of the above-mentioned business process graph, the concepts of information dependence of various types are formalized. The proposed work is one of the first attempts to build a formal basis for the organization of parallelism in business processes, which plays an important role in improving the key indicators of business processes in the tasks of their reorganization.

LOSP: Overlap Synchronization Parallel With Local Compensation for Fast Distributed Training

When running in Parameter Server (PS), the Distributed Stochastic Gradient Descent (D-SGD) incurs significant communication delays and huge communication overhead due to the model synchronization. Moreover, considering the heterogeneity of computational capability among workers, traditional synchronization modes incur under-utilization of computational resources because fast workers have to wait for slow ones finishing the computation. Although our previous work OSP can effectively solve these problems by overlapping the computation and communication procedures and allowing adaptive multiple local updates in distributed training, it causes the staleness problem brought by the overlap, yielding a performance degradation. In this paper, we propose a new method named LOSP by introducing local compensation to our previous synchronization mechanism, which mitigates adverse effects caused by the overlapping synchronization. We theoretically prove that LOSP (1) preserves the same convergence rate as the sequential SGD for non-convex problems, and (2) exhibits good scalability due to the linear speedup property with respect to both the number of workers and the average number of local updates. Evaluations show that LOSP significantly improves performance over the state-of-the-art ones in terms of both convergence accuracy and communication cost.

Partial Synchronization to Accelerate Federated Learning Over Relay-Assisted Edge Networks

Federated Learning (FL) is a promising machine learning paradigm to cooperatively train a global model with highly distributed data located on mobile devices. Aiming to optimize the communication efficiency for gradient aggregation and model synchronization among large-scale devices, we propose a relay-assisted FL framework. By breaking the traditional transmission-order constraint and exploiting the broadcast characteristic of relay nodes, we design a novel synchronization scheme named Partial Synchronization Parallel (PSP), in which models and gradients are transmitted simultaneously and aggregated at relay nodes, resulting in traffic reduction. We prove that PSP has the same convergence rate as the sequential synchronization approaches via rigorous analysis. To further accelerate the training process, we integrate PSP with any unbiased and error-bounded compression technologies and prove that the convergence properties of the resulting scheme still hold. Extensive experiments are conducted in a distributed cluster environment with real-world datasets and the results demonstrate that our proposed approach reduces the training time up to 37 percent compared to state-of-the-art methods.