Asynchronous Iterations Research Articles

APapo: An asynchronous parallel optimization method for DNN models

To address the challenges related to segmentation complexity, high memory usage, extended training duration, and low equipment utilization in parallel optimization of large-scale deep neural network (DNN) models, this paper proposes an asynchronous parallel optimization method APapo. Firstly, a multi-iteration asynchronous pipeline parallel scheduling was established for model parallel computing tasks, controlling the specific scheduling process of micro-batch units to address gradient delay updating during asynchronous iteration. Secondly, combined with the given network model and hardware configuration, a dynamic programming strategy for computing resources and model tasks was designed to achieve dynamic segmentation of model computing tasks and optimal matching of computing resources. Finally, an optimization strategy for runtime scheduling of computing resources and model tasks was developed, using improved device streams to maximize the overlap between computing and communication, thus improving the utilization rate of computing resources and reducing training time. Experimental results show that the APapo method achieves fine-grained task segmentation, maximizes the utilization rate of each GPU computing resource, and on average improves the training speed of large-scale deep neural network models by 2.8 times while maintaining the training accuracy of the model compared to existing parallel optimization methods.

A Performance of AFIRO among Asynchronous Iteration Strategy Metaheuristic Algorithms

Asynchronous Finite Impulse Response Optimizer (AFIRO) is a metaheuristic algorithm that has been developed as a population-based solution with an asynchronous update mechanism. AFIRO is inspired by the Ultimate Unbiased Finite Impulse Response filter framework. AFIRO works with a group of agents where each agent performs the iteration update asynchronously. In the original paper, AFIRO was compared with the Particle Swarm Optimisation algorithm, Genetic Algorithm, and Grey Wolf Optimizer. Although AFIRO shows a better performance, the comparison seems unfair since the iteration strategy of AFIRO is different from those compared algorithms. Hence, this article further investigates the potential of AFIRO against three existent metaheuristic algorithms with the same iteration strategy, namely Asynchronous PSO (A-PSO), Asynchronous Gravitational Search Algorithm (A-GSA), and Asynchronous Simulated Kalman Filter (A-SKF). The CEC2014 test suite was applied to evaluate the performance, where the results revealed that AFIRO leads 18 out of 30 functions. The Holm post hoc showed that AFIRO performs significantly better than A-SKF and A-GSA while having the same performance as A- PSO. Moreover, the Friedman test disclosed that AFIRO has the highest ranking than A-PSO, A-SKF, and A-GSA. Therefore, it can be concluded that AFIRO performs well in the same iteration strategy category.

Asynchronous global–local non-invasive coupling for linear elliptic problems

This paper presents the first asynchronous version of the Global/Local non-invasive coupling, capable of dealing efficiently with multiple, possibly adjacent, patches. We give a new interpretation of the coupling in terms of primal domain decomposition method, and we prove the convergence of the relaxed asynchronous iteration. The asynchronous paradigm lifts many bottlenecks of the Global/Local coupling performance. We illustrate the method on several linear elliptic problems as encountered in thermal and elasticity studies.

AAFL: Asynchronous-Adaptive Federated Learning in Edge-Based Wireless Communication Systems for Countering Communicable Infectious Diseasess

With the rapid growth of the coronavirus disease of 2019 (COVID-19) cases, massive amounts of relevant data are being trained on machine learning models for countering communicable infectious diseases. Federated Learning (FL) is a paradigm of distributed machine learning to deal with the individual COVID-19 data, and enable the protection of data privacy. However, FL has low efficiency in Edge-Based wireless communication systems with system heterogeneity. In this paper, we propose an “Asynchronous-Adaptive FL” (AAFL) scheme. Specifically, we allow that medical devices with different performances have a heterogeneous number of local SGD iterations in each communication round, called asynchronous iteration strategy which is balanced under adaptive control. We theoretically analyze the convergence of the AAFL scheme under a given time budget and obtain a mathematical relationship between the heterogeneous number of local SGD iterations and the optimal model parameters. Based on the mathematical relationship, we design an algorithm for parameter server and work nodes to adaptively control the heterogeneous number of local SGD iterations. Subsequently, we build a prototype heterogeneous system and conduct experiments on various scenarios for analyzing the general properties of our algorithm, and then apply our algorithm to public COVID-19 databases. The experimental results and application performance demonstrate the effectiveness and efficiency of our AAFL scheme.

Scheduling Hyperparameters to Improve Generalization: From Centralized SGD to Asynchronous SGD

This paper 1 studies how to schedule hyperparameters to improve generalization of both centralized single-machine stochastic gradient descent (SGD) and distributed asynchronous SGD (ASGD). SGD augmented with momentum variants (e.g., heavy ball momentum (SHB) and Nesterov’s accelerated gradient (NAG)) has been the default optimizer for many tasks, in both centralized and distributed environments. However, many advanced momentum variants, despite empirical advantage over classical SHB/NAG, introduce extra hyperparameters to tune. The error-prone tuning is the main barrier for AutoML. Centralized SGD : We first focus on centralized single-machine SGD and show how to efficiently schedule the hyperparameters of a large class of momentum variants to improve generalization. We propose a unified framework called multistage quasi-hyperbolic momentum (Multistage QHM), which covers a large family of momentum variants as its special cases (e.g. vanilla SGD/SHB/NAG). Existing works mainly focus on only scheduling learning rate α ’s decay, while multistage QHM allows additional varying hyperparameters (e.g., momentum factor), and demonstrates better generalization than only tuning α . We show the convergence of multistage QHM for general nonconvex objectives. Distributed SGD : We then extend our theory to distributed asynchronous SGD (ASGD), in which a parameter server distributes data batches to several worker machines and updates parameters via aggregating batch gradients from workers. We quantify the asynchrony between different workers (i.e., gradient staleness), model the dynamics of asynchronous iterations via a stochastic differential equation (SDE), and then derive a PAC-Bayesian generalization bound for ASGD. As a byproduct, we show how a moderately large learning rate helps ASGD to generalize better. Our tuning strategies have rigorous justifications rather than a blind trial-and-error as we theoretically prove why our tuning strategies could decrease our derived generalization errors in both cases. Our strategies simplify the tuning process and beat competitive optimizers in test accuracy empirically. Our codes are publicly available https://github.com/jsycsjh/centralized-asynchronous-tuning.

Resilient asynchronous primal Schur method

This paper introduces the application of asynchronous iterations theory within the framework of the primal Schur domain decomposition method. A suitable relaxation scheme is designed, whose asynchronous convergence is established under classical spectral radius conditions. For the usual case where local Schur complement matrices are not constructed, suitable splittings based only on explicitly generated matrices are provided. Numerical experiments are conducted on a supercomputer for both Poisson’s and linear elasticity problems. The asynchronous Schur solver outperformed the classical conjugate-gradient-based one in case of computing node failures.

Dynamic asynchronous iterations

Many problems can be solved by iteration by multiple participants (processors, servers, routers etc.). Previous mathematical models for such asynchronous iterations assume a single function being iterated by a fixed set of participants. We will call such iterations static since the system's configuration does not change. However in several real-world examples, such as inter-domain routing, both the function being iterated and the set of participants change frequently while the system continues to function. In this paper we extend Üresin and Dubois's work on static iterations to develop a model for this class of dynamic or always on asynchronous iterations. We explore what it means for such an iteration to be implemented correctly, and then prove two different conditions on the set of iterated functions that guarantee the full asynchronous iteration satisfies this new definition of correctness. These results have been formalised in Agda and the resulting library is publicly available.

Fast datalog evaluation for batch and stream graph processing

Implementing complex algorithms for big data, artificial intelligence, and graph processing requires enormous effort. Succinct, declarative programs to solve complex problems that can be efficiently executed for batching and streaming data are in demand. This paper presents Nexus, a distributed Datalog evaluation system. It evaluates Datalog programs using the semi-naive algorithm for batch and streaming data using incremental and asynchronous iteration. Furthermore, we evaluate Datalog programs with aggregates to determine the advantages of implementing the semi-naive algorithm using incremental iteration on its performance. Our experimental results show that Nexus significantly outperforms acyclic dataflow-based systems.

Optimization of Simultaneous Delivery and Pickup Wagon Scheme on Hybrid Siding Network of Railway Terminal

Aiming at the local freight train transit system in railway terminal, the optimization of simultaneous delivery and pickup wagon scheme on hybrid siding network is presented. The problem is formulated as a mathematical programming model. The objective function intends to minimize shunting engines’ operating cost, wagons’ travelling cost, and penalty cost for exceeding the retrieval time window. Some constraints indicating the relationship between delivery and pickup wagons, relationship between radial branches and operation batches, relationship between local wagon groups’ retrieved time and outbound train departure time, and relationship between tasks and operation batches are considered in the model. As an NP problem, using traditional method to solve the model is difficult and inefficient. A novel two‐stage hybrid optimization procedure is proposed. Firstly, the three‐phase approach composed of coding task sequence, dividing task batch, and generating access order sequence population is developed to generate initial solution. Secondly, an asynchronous iteration heuristic is provided. The iteration object is given as the expression form of the solution representing the delivery and pickup wagon sequence and then the initial iteration population is generated. Furthermore, the primary updating iteration population with pending and remaining groups is introduced to optimize iteration object Then, the senior updating iteration population with Lévy flight searching is also introduced to optimize further iteration object. Finally, the experimental scenarios are designed to test the proposed approach. Also, the proposed approach is compared with some other approaches and the performance of the proposed approach is evaluated by some different sized instances.

Self-Identification of Cable-Driven Exoskeleton Based on Asynchronous Iterative Method

Abstract The upper limb rehabilitation exoskeleton with cable-driven parallel structure has the advantages of light weight and large payload, etc. However, due to the non-rigid nature of the actuating cables and the different body shape of the wearer, the geometric parameters of the exoskeleton have a large error. The parameter identification of cable-driven exoskeleton is of great significance. An asynchronous self-identification method for the upper limb seven degree-of-freedom (DOF) cable-driven exoskeleton was proposed and used in a wearable multi-redundant exoskeleton. Asynchronous iteration eliminates the accumulation of joint errors. High identification reliability is achieved by selecting proper identification parameters and optimizing error model.With the method, the geometric parameters of the exoskeleton can be identified by using exoskeleton joint angle and cable length data. The experiment verifies that the success rate of parameter identification for different wearers is in line with expectations, and the control precision and stability of the prototype are greatly improved after parameter identification.

Asynchronous iterations of HSS method for non-Hermitian linear systems

A general asynchronous alternating iterative model is designed, for which convergence is theoretically ensured both under classical spectral radius bound and, then, for a classical class of matrix splittings for -matrices. The computational model can be thought of as a two-stage alternating iterative method, which well suits to the well-known Hermitian and skew-Hermitian splitting (HSS) approach, with the particularity here of considering only one inner iteration. Experimental parallel performance comparison is conducted between the generalized minimal residual (GMRES) algorithm, the standard HSS and our asynchronous variant, on both real and complex non-Hermitian linear systems, respectively, arising from convection–diffusion and structural dynamics problems. A significant gain on execution time is observed in both cases.

Asynchronous substructuring method with alternating local and global iterations

Until now, almost all investigations of asynchronous iterations within domain decomposition frameworks targeted methods of the parallel Schwarz type. A first, and sole, attempt to deal with a primal substructuring framework resulted in an asynchronous substructuring method where relaxation occurs simultaneously on the subdomains and on the interface between them, which therefore corresponds to a substructured relaxation scheme defined on the whole global domain. In this paper, we propose a Gauss–Seidel kind of improvement consisting of alternating between relaxation on the interface and relaxation on the subdomains, hence, always using the latest solutions in the subdomains when updating the solution on the interface, which is feasible at no additional cost. It turns out that one particular case of our general alternating relaxation scheme corresponds to an asynchronous substructuring method with iterations fully defined on the subdomains’ interface, and where only local Schur complements are involved. Practical performance evaluation on both standard Poisson’s and linear elasticity problems has been conducted using a multi-node parallel computational platform with up to 720 CPU cores.

Point-block incomplete LU preconditioning with asynchronous iterations on GPU for multiphysics problems

Point-block matrices arise naturally in multiphysics problems when all variables associated with a mesh point are ordered together, and are different from the general block matrices since the sizes of the blocks are so small one can often invert some of the diagonal blocks explicitly. Motivated by the recent works of Chow and Patel and Chow et al., we propose an efficient incomplete LU (ILU) preconditioner for point-block matrices targeting applications on GPU. The construction of the preconditioner involves two critical steps: (1) the initial guessing of values for the lower and upper triangular matrices; and (2) several sweeps of asynchronous updating of the triangular matrices. Three representative problems are studied to show the advantage of the proposed point-block approach over the standard point-wise approach in terms of the number of GMRES iterations and also the total compute time. Moreover, we compare the proposed algorithm with the level-scheduling based parallel algorithm employed in NVIDIA’s cuSPARSE library as well as the serial method implemented in Intel MKL library, and the experiments show that a 2×–5× speedup can be achieved over the block-based ILU( p) factorizations from the cuSPARSE library.

Asynchronous Distributed Power Control of Multimicrogrid Systems

Asynchrony widely exists in microgrids (MGs), such as nonidentical sampling rates and communication delays, which challenges the MG control. This article addresses the asynchronous distributed power control problem of hybrid microgrids, considering different kinds of asynchrony, such as nonidentical sampling rates, and random time delays. To this end, we first formulate the economic dispatch problem of MGs, and devise a synchronous algorithm. Then, we analyze the impact of asynchrony, and propose an asynchronous iteration algorithm based on the synchronous version. By introducing a random clock at each iteration, different types of asynchrony are fitted into a unified framework, where the asynchronous algorithm is converted into a fixed-point iteration problem with a nonexpansive operator, leading to a convergence proof. We further provide an upper bound estimation of the time delay. Moreover, the real-time implementation of the proposed algorithm in both ac and dc MGs is introduced. By measuring the frequency/voltage, the controller is simplified by reducing one order, and adapting to the fast varying load demand. Finally, simulations on a benchmark MG, and experiments are utilized to verify the effectiveness, and advantages of the proposed algorithm.

Fabrication of Amorphous SRVOX-Based Memristor for Neuromorphic Computing

Because resistive random access memory (RRAM) has advantages, such as fast switching speed, low energy consumption, and simple device structure, it is promising device architecture for memory, artificial intelligence, and neuromorphic computing. In particular, for RRAM for use in neuromorphic applications, it must be able to mimic synaptic behavior by pulse modulation. In order to have such a characteristic, the resistance state has to be gradually changed by the stress, e.g., potentiation and depression. High linearity is required because the linearity of RRAM obtained through each resistance state is evaluated as an especially important property for artificial synaptic applications.In this work, we performed the electrical properties of sputter-deposited amorphous strontium vanadate (a-SVO) RRAM with Pt bottom electrode and Ag top electrode. Sufficient reproducibility and uniformity between devices was realized. The highest resistance ratio of ~ 103 was observed and insignificantly changed for retention testing up to 2 × 104 s. In particular, the conduction filament generated from the a-SVO RRAM was adjusted according to the regulation of voltage stress to obtain a multi-level resistance state. It was confirmed by ToF-SIMS analysis and COMSOL Multiphysics simulation that the conduction filament was formed by Ag atoms. Hopfield Neural Network simulation was conducted to verify that the performance of analog resistive switching behavior of a-SVO RRAM is suitable for neuromorphic computing. The accuracy of our simulated device was saturated to 86.8 % for 100,000 asynchronous iterations. The results imply that our a-SVO RRAM is of great potential for use in neuromorphic computing.

Convergence versus Divergence Behaviors of Asynchronous Iterations, and Their Applications in Concrete Situations

Asynchronous iterations have long been used in distributed computing algorithms to produce calculation methods that are potentially faster than a serial or parallel approach, but whose convergence is more difficult to demonstrate. Conversely, over the past decade, the study of the complex dynamics of asynchronous iterations has been initiated and deepened, as well as their use in computer security and bioinformatics. The first work of these studies focused on chaotic discrete dynamical systems, and links were established between these dynamics on the one hand, and between random or complex behaviours in the sense of the theory of the same name. Computer security applications have focused on pseudo-random number generation, hash functions, hidden information, and various security aspects of wireless sensor networks. At the bioinformatics level, this study of complex systems has allowed an original approach to understanding the evolution of genomes and protein folding. These various contributions are detailed in this review article, which is an extension of the paper “An update on the topological properties of asynchronous iterations” presented during the Sixth International Conference on Parallel, Distributed, GPU and Cloud Computing (Pareng 2019).

Energy efficiency of nonlinear domain decomposition methods

A nonlinear domain decomposition (DD) solver is considered with respect to improved energy efficiency. In this method, nonlinear problems are solved using Newton’s method on the subdomains in parallel and in asynchronous iterations. The method is compared to the more standard Newton-Krylov approach, where a linear domain decomposition solver is applied to the overall nonlinear problem after linearization using Newton’s method. It is found that in the nonlinear domain decomposition method, making use of the asynchronicity, some processor cores can be set to sleep to save energy and to allow better use of the power and thermal budget. Energy savings on average for each socket up to 77% (due to the RAPL hardware counters) are observed compared to the more traditional Newton-Krylov approach, which is synchronous by design, using up to 5120 Intel Broadwell (Xeon E5-2630v4) cores. The total time to solution is not affected. On the contrary, remaining cores of the same processor may be able to go to turbo mode, thus reducing the total time to solution slightly. Last, we consider the same strategy for the ASPIN (Additive Schwarz Preconditioned Inexact Newton) nonlinear domain decomposition method and observe a similar potential to save energy.

Protocol-free asynchronous iterations termination

In this paper, we tackled the convergence detection problem arisen from the absence of synchronization during asynchronous iterative computation. We showed that, when one arbitrarily takes the local components of a global solution vector, an upper bound can be established on the difference between a residual error evaluated from this global vector and the inconsistent residual error evaluated without synchronizing the involved computing processes. This allows for accurate termination of asynchronous iterations without implementing any particular detection protocol. Termination delay has be handled too for not slowing down the overall asynchronous solver, by appropriately setting the convergence threshold criterion. We therefore ensured effectiveness while reaching better efficiency in terms of overall execution time of the solver, in comparison with the current most efficient exact snapshot-based approach.

Maximum sensitivity to update schedules of elementary cellular automata over infinite configurations

Cellular automata are discrete dynamical systems with locally defined behaviour, well known as simple models of complex systems. Classically, their dynamics derive from synchronously iterated rules over finite or infinite configurations; however, since for many natural systems to be modelled, asynchrony seems more plausible, asynchronous iteration of the rules has gained a considerable attention in recent years. One question in this context is how changing the update schedule of rule applications affects the global behaviour of the system. In particular, previous works addressed the notion of maximum sensitivity to changes in the update schemes for finite lattices. Here, we extend the notion to infinite lattices, and classify elementary cellular automata space according to such a property.

Random asynchronous iterations in distributed coordination algorithms

Distributed coordination algorithms (DCA) carry out information processing processes among a group of networked agents without centralized information fusion. Though it is well known that DCA characterized by an SIA (stochastic, indecomposable, aperiodic) matrix generate consensus asymptotically via synchronous iterations, the dynamics of DCA with asynchronous iterations have not been studied extensively, especially when viewed as stochastic processes. This paper aims to show that for any given irreducible stochastic matrix, even non-SIA, the corresponding DCA lead to consensus successfully via random asynchronous iterations under a wide range of conditions on the transition probability. Particularly, the transition probability is neither required to be i.i.d, nor characterized by a Markov chain.