Inter-node Communication Cost Research Articles

Resilient edge machine learning in smart city environments

Distributed Machine Learning (DML) has emerged as a disruptive technology that enables the execution of Machine Learning (ML) and Deep Learning (DL) algorithms in proximity to data generation, facilitating predictive analytics services in Smart City environments. However, the real-time analysis of data generated by Smart City Edge Devices (EDs) poses significant challenges. Concept drift, where the statistical properties of data streams change over time, leads to degraded prediction performance. Moreover, the reliability of each computing node directly impacts the availability of DML systems, making them vulnerable to node failures. To address these challenges, we propose a resilience framework comprising computationally lightweight maintenance strategies that ensure continuous quality of service and availability in DML applications. We conducted a comprehensive experimental evaluation using real datasets, assessing the effectiveness and efficiency of our resilience maintenance strategies across three different scenarios. Our findings demonstrate the significance and practicality of our framework in sustaining predictive performance in smart city edge learning environments. Specifically, our enhanced model exhibited increased generalizability when confronted with concept drift. Furthermore, we achieved a substantial reduction in the amount of data transmitted over the network during the maintenance of the enhanced models, while balancing the trade-off between the quality of analytics and inter-node data communication cost.

Cooperative Terrain Navigation Using Hybrid GMM/SMC Message Passing on Factor Graphs

We introduce in this article a distributed factor-graph-based algorithm for anchorless cooperative aircraft localization in a GNSS-denied scenario without fixed infrastructure. The agents use terrain-aided navigation (TAN) to perform local position estimation and exchange messages to improve their position beliefs. Internode communication cost is reduced using a hybrid Gaussian mixture model / sequential Monte Carlo (GMM/SMC) approach. Simulation results show that, even in a partially connected network where only a small part of the agents perform TAN, cooperation yields better results than all aircraft performing TAN independently.

Diffusion self-triggered square-root cubature information filter for nonlinear non-Gaussian systems and its application to the optic-electric sensor network

This article focuses on the problem of the distributed estimation for a class of discrete-time nonlinear non-Gaussian systems. In this paper, the non-Gaussian noises are approximated by the Gaussian mixture model. A novel method to merge the Gaussian components has been proposed in this article to improve its numerical stability. A diffusion Gaussian mixture square-root cubature information filter has been proposed in this paper. To ease the internode communication cost further, we design a self-triggered mechanism in this article. With the proposed mechanism, each sensor node transmits its information to its neighbors only when it’s necessary. The effectiveness of the proposed algorithm is verified through its application to an optic-electric sensor network.

Communication-Efficient Decentralized Sparse Bayesian Learning of Joint Sparse Signals

We consider the problem of decentralized estimation of multiple joint sparse vectors by a network of nodes from locally acquired noisy and underdetermined linear measurements, when the cost of communication between the nodes is at a premium. We propose an iterative, decentralized Bayesian algorithm called fusion-based distributed sparse Bayesian learning (FB-DSBL) in which the nodes collaborate by exchanging highly compressed messages to learn a common joint sparsity inducing signal prior. The learnt signal prior is subsequently used by each node to compute the maximum a posteriori probability estimate of its respective sparse vector. Since the internode communication cost is expensive, the size of the messages exchanged between nodes is reduced substantially by exchanging only those local signal prior parameters which are associated with the nonzero support detected via multiple composite log-likelihood ratio tests. The average message size is empirically shown to be proportional to the information rate of the unknown vectors. The proposed sparse Bayesian learning (SBL)-based distributed algorithm allows nodes to exploit the underlying joint sparsity of the signals. In turn, this enables the nodes to recover sparse vectors with significantly lower number of measurements compared to the standalone SBL algorithm. The proposed algorithm is interpreted as a degenerate case of a distributed consensus-based stochastic approximation algorithm for finding a fixed point of a function, and its generalized version with Robbins–Monro-type iterations is also developed. Using Monte Carlo simulations, we demonstrate that the proposed FB-DSBL has superior mean squared error and support recovery performance compared to the existing decentralized algorithms with similar or higher communication complexity.

A novel cooperative accelerated parallel two-list algorithm for solving the subset-sum problem on a hybrid CPU–GPU cluster

Many parallel algorithms have recently been developed to accelerate solving the subset-sum problem on a heterogeneous CPU–GPU system. However, within each compute node, only one CPU core is used to control one GPU and all the remaining CPU cores are in idle state, which leads to a large number of CPU cores being wasted. In this paper, based on a cost-optimal parallel two-list algorithm, we propose a novel heterogeneous cooperative computing approach to solve the subset-sum problem on a hybrid CPU–GPU cluster, which can make full use of all available computational resources of a cluster. The unbalanced workload distribution and the huge communication overhead are two main obstacles for the heterogeneous cooperative computing. In order to assign the most suitable workload to each compute node and reasonably partition it between CPU and GPU within each node, and minimize the inter-node and intra-node communication costs, we design a communication-avoiding workload distribution scheme suitable for the parallel two-list algorithm. According to this scheme, we provide an efficient heterogeneous cooperative implementation of the algorithm. A series of experiments are conducted on a hybrid CPU–GPU cluster, where each node has two 6-core CPUs and one GPU. The results show that the heterogeneous cooperative computing significantly outperforms the CPU-only or GPU-only computing.

Achieving Optimal Inter-Node Communication in Graph Partitioning Using Random Selection and Breadth-First Search

Processing large graph datasets represents an increasingly important area in computing research and applications. The size of many graph datasets has increased well beyond the processing capacity of a single computing node, thereby necessitating distributed approaches. As these datasets are processed over a distributed system of nodes, this leads to an inter-node communication cost problem that negatively affects system performance. Previously proposed algorithms implemented breadth-first search (BFS) for graph searching and focused on the execution, parallel performance and not the communication. In this paper a new methodology is proposed that combines BFS with random selection in order to partition large graph datasets and effectively minimize inter-node communication. The new method is discussed and applied to the single-source shortest path and PageRank algorithms using three graphs that are representative of real-world scenarios. Experimental results show that graph inter-node communication for canonical graphs representative of real-world data is improved up to 42 % in case of Powerlaw graph, up to 27 % in case of Random near K-regular graph (with low degree), and up to 7 % in case of Random near K-regular graph (with high degree).

Improved strong scaling of a spectral/finite difference gyrokinetic code for multi-scale plasma turbulence

Abstract Optimization techniques of a plasma turbulence simulation code GKV for improved strong scaling are presented. This work is motivated by multi-scale plasma turbulence extending over multiple spatio-temporal scales of electrons and ions, whose simulations based on the gyrokinetic theory require huge calculations of five-dimensional (5D) computational fluid dynamics by means of spectral and finite difference methods. First, we present the multi-layer domain decomposition of the multi-dimensional and multi-species problem, and segmented MPI-process mapping on 3D torus interconnects, which fully utilizes the bi-section bandwidth for data transpose and reduces the conflicts of simultaneous point-to-point communications. These techniques reduce the inter-node communication cost drastically. Second, pipelined computation-communication overlaps are implemented by using the OpenMP/MPI hybrid parallelization, which effectively mask the communication cost. Careful regulations of the pipeline length and of the thread granularity respectively suppress latencies of MPI, load imbalance and scheduling overheads of OpenMP. Thanks to the above optimizations, GKV achieves excellent strong scaling up to ∼600k cores with high computational performance 782.4 TFlops (8.29% of the theoretical peak) and high effective parallelization rate ∼99.99994% on K, which demonstrates its applicability and efficiency toward a million of cores. The optimized code realizes multi-scale plasma turbulence simulations covering electron and ion scales, and reveals cross-scale interactions of electron- and ion-scale turbulence.

An adaptive and hierarchical task scheduling scheme for multi-core clusters

• An adaptive and hierarchical task scheduling scheme (AHS) is proposed. • Work-sharing is used in conjunction with work-stealing. • An initial partitioning is performed with respect to the pattern of task parallelism . • A practical implementation of AHS is described. • The theoretical, simulation and experimental studies of AHS are presented. Work-stealing and work-sharing are two basic paradigms for dynamic task scheduling. This paper introduces an adaptive and hierarchical task scheduling scheme (AHS) for multi-core clusters, in which work-stealing and work-sharing are adaptively used to achieve load balancing. Work-stealing has been widely used in task-based parallel programing languages and models, especially on shared memory systems. However, high inter-node communication costs hinder work-stealing from being directly performed on distributed memory systems. AHS addresses this issue with the following techniques: (1) initial partitioning, which reduces the inter-node task migrations; (2) hierarchical scheduling scheme, which performs work-stealing inside a node before going across the node boundary and adopts work-sharing to overlap computation and communication at the inter-node level; and (3) hierarchical and centralized control for inter-node task migration, which improves the efficiency of victim selection and termination detection. We evaluated AHS and existing work-stealing schemes on a 16-nodes multi-core cluster. Experimental results show that AHS outperforms existing schemes by 11–21.4%, for the benchmarks studied in this paper.

Collaborative emitter tracking using Rao-Blackwellized random exchange diffusion particle filtering

We introduce in this paper the fully distributed, random exchange diffusion particle filter (ReDif-PF) to track a moving emitter using multiple received signal strength (RSS) sensors. We consider scenarios with both known and unknown sensor model parameters. In the unknown parameter case, a Rao-Blackwellized (RB) version of the random exchange diffusion particle filter, referred to as the RB ReDif-PF, is introduced. In a simulated scenario with a partially connected network, the proposed ReDif-PF outperformed a PF tracker that assimilates local neighboring measurements only and also outperformed a linearized random exchange distributed extended Kalman filter (ReDif-EKF). Furthermore, the novel ReDif-PF matched the tracking error performance of alternative suboptimal distributed PFs based respectively on iterative Markov chain move steps and selective average gossiping with an inter-node communication cost that is roughly two orders of magnitude lower than the corresponding cost for the Markov chain and selective gossip filters. Compared to a broadcast-based filter which exactly mimics the optimal centralized tracker or its equivalent (exact) consensus-based implementations, ReDif-PF showed a degradation in steady-state error performance. However, compared to the optimal consensus-based trackers, ReDif-PF is better suited for real-time applications since it does not require iterative inter-node communication between measurement arrivals.

Complexity Analysis of New Task Allocation Problem Using Network Flow Method on Multicore Clusters

The task allocation problem (TAP) generally aims to minimize total execution cost and internode communication cost in traditional parallel computing systems. New TAP (NTAP) considering additive intranode communication cost in emerging multicore cluster systems is investigated in this paper. We analyze the complexity of NTAP with network flow method and conclude that the intranode communication cost is a key to the complexity of NTAP, and prove that (1) the NTAP can be cast as a generalized linear network minimum cost flow problem and can be solved inO(m2n4)time if the intranode communication cost equals the internode communication cost, and (2) the NTAP can be cast as a generalized convex cost network minimum cost flow problem and can be solved in polynomial time if the intranode communication cost is more than the internode communication cost. More in particular, the uniform cost NTAP can be cast as a convex cost flow problem and can be solved inO(m2n2log(m+n))time. Furthermore, solutions to the NTAP are also discussed. Our work extends currently known theoretical results and the theorems and conclusions presented in this paper can provide theoretical basis for task allocating strategies on multicore clusters.

Layout-aware scientific computing: A case study using the MILC code

Nowadays, high performance computers have more cores and nodes than ever before. Computation is spread out among them, leading to more communication cost than before. For this reason, communication can easily become the bottleneck of a system and limit its scalability. The layout of an application on a computer is the key factor to preserve communication locality and reduce its cost. In this paper, we propose a straightforward model to optimize the layout for scientific applications by minimizing inter-node communication cost. The model takes into account the latency and bandwidth of the network and associates them with the dominant layout variables of the application. We take the MILC code as an example and analyze its communication patterns. According to our experimental results, the model developed for the MILC code achieved a satisfactory accuracy for predicting the performance, leading to up to 31% performance improvement.

Hybrid Decomposition Method in Parallel Molecular Dynamics Simulation Based on SMP Cluster Architecture

A hybrid decomposition method for molecular dynamics simulations was presented, using simultaneously spatial decomposition and force decomposition to fit the architecture of a cluster of symmetric multi-processor (SMP) nodes. The method distributes particles between nodes based on the spatial decomposition strategy to reduce inter-node communication costs. The method also partitions particle pairs within each node using the force decomposition strategy to improve the load balance for each node. Simulation results for a nucleation process with 4 000 000 particles show that the hybrid method achieves better parallel performance than either spatial or force decomposition alone, especially when applied to a large scale particle system with non-uniform spatial density.

Distribution algorithms for document allocation in multiprocessor information retrieval systems

Distributed memory information retrieval systems have been used as a means of managing the vast volume of documents in an information retrieval system, and to improve query response time. However, proper allocation of documents plays an important role in improving the performance of such systems. Maximising the amount of parallelism can be achieved by distributing the documents, while the inter-node communication cost is minimised by avoiding documents distribution. Unfortunately, these two factors contradict each other. Finding an optimal allocation satisfying the above objectives is referred to as distributed memory document allocation problem (DDAP), and it is an NP-Complete problem. Heuristic algorithms are usually employed to find an optimal solution to this problem. Genetic algorithm is one such algorithms. In this paper, a genetic algorithm is developed to find an optimal document allocation for DDAP. Several well-known network topologies are investigated to evaluate the performance of the algorithm. The approach relies on the fact that documents of an information retrieval system are clustered by some arbitrary method. The advantages of a clustered document approach specially in a distributed memory information retrieval system are well-known. Since genetic algorithms work with a set of candidate solutions, parallelisation based on a Single Instruction Multiple Data (SIMD) paradigm seems to be the natural way to obtain a speedup. Using this approach, the population of strings is distributed among the processing elements. Each string is processed independently. The performance gain comes from the parallel execution of the strings, and hence, it is heavily dependent on the population size. The approach is favoured for genetic algorithms' applications where the parameter set for a particular run is well-known in advance, and where such applications require a big population size to solve the problem. DDAP fits nicely into the above requirements. The aim of the parallelisation is two-fold: the first one is to speedup the allocation process in DDAP which usually consists of thousands of documents and has to use a big population size, and second, it can be seen as an attempt to port the genetic algorithm's processes into SIMD machines.