Inter-node Communication Research Articles

Implementation of a gyrokinetic collision operator with an implicit time integration scheme and its computational performance

We have implemented the Sugama collision operator in the gyrokinetic Vlasov simulation code, GKV, with an implicit time-integration scheme. The new method is versatile and independent of the details of the linearized collision operator, by means of an operator splitting, an implicit time integrator, and an iterative Krylov subspace solver. Numerical tests demonstrate stable computation over the time step size restricted by the collision term. An efficient implementation for parallel computation on distributed memory systems is realized by using the data transpose communication, which makes the iterative solver free from inter-node communications during iteration. Consequently, the present approach achieves enhancement of computational efficiency and reduction of computational time to solution simultaneously, and significantly accelerates the total performance of the application.

Efficient techniques for energy saving in data center networks

Data centers are constructed with a huge number of network devices to support the expanding cloud based services. These devices are used to achieve the highest performance in case of full utilization of the network. However, the peak capacity of the network is rarely reached. Consequently, many devices are set into idle state and cause a huge energy waste leading to a non-proportionality between the network load and the energy consumed. In this paper, we propose a new approach to improve the efficiency of data centers in terms of energy consumption. Our approach exploits the correlation in time of the inter-node communication traffic and some topological features to maximize energy saving with only a minor increase in the average path length. Our approach dynamically controls the number of active communication links by turning off and on ports in the network (switches ports and nodes ports). Simulations results confirmed the energy saving gain procured by the proposed approach with a low impact on the average path length.

Efficient distributed memory management with RDMA and caching

Recent advancements in high-performance networking interconnect significantly narrow the performance gap between intra-node and inter-node communications, and open up opportunities for distributed memory platforms to enforce cache coherency among distributed nodes. To this end, we propose GAM, an efficient distributed in-memory platform that provides a directory-based cache coherence protocol over remote direct memory access (RDMA). GAM manages the free memory distributed among multiple nodes to provide a unified memory model, and supports a set of user-friendly APIs for memory operations. To remove writes from critical execution paths, GAM allows a write to be reordered with the following reads and writes, and hence enforces partial store order (PSO) memory consistency. A light-weight logging scheme is designed to provide fault tolerance in GAM. We further build a transaction engine and a distributed hash table (DHT) atop GAM to show the ease-of-use and applicability of the provided APIs. Finally, we conduct an extensive micro benchmark to evaluate the read/write/lock performance of GAM under various workloads, and a macro benchmark against the transaction engine and DHT. The results show the superior performance of GAM over existing distributed memory platforms.

NSTX-U advances in real-time deterministic PCIe-based internode communication

Distributing control mechanisms across modern commercial off the shelf (COTS) GNU/Linux systems often introduces difficult to mitigate non-deterministic behavior. Existing methods to address this problem involve non-real-time technologies such as Remote Direct Memory Access (RDMA) over Infiniband or custom Ethernet solutions, such as RDMA over Converged Ethernet (RoCE) or userspace Ethernet drivers, that trade determinism for ease of use or lower cost. The National Spherical Torus Experiment Upgrade (NSTX-U) is pursuing a new design that allows direct communication between heterogeneous systems with scalable, microsecond latency with 1 μs of jitter on that latency, outside of the constant transmission delay at the physical layer. The future design of the NSTX-U Real-time Communication System will utilize direct PCIe-to-PCIe communication with kernel support tuned for low overhead, allowing two (or more, through a switch) real-time (RT) systems to communicate and share resources as one larger entity. This greatly increases the processing capability of the primary Plasma Control System (PCS), turning previously insurmountable computational challenges into a more manageable divide and conquer parallel task.

Stability and active power sharing in droop controlled inverter interfaced microgrids: Effect of clock mismatches

Stability and power sharing properties of droop controlled inverter-based microgrid systems depend on various design factors. Little explored is the effect of component mismatches and parameters drifts on the stability, steady state behaviour and power sharing properties of these systems. In this paper, the behaviour of frequency droop controlled inverter based microgrid systems in the presence of non-identical clocks is analysed. It is shown that power sharing between converters in a microgrid can be sensitive to clock mismatches. Our proposal shows that a coordination control that uses sparse inter-node communications is useful in ensuring desired active power sharing. Conditions are derived to ensure stability in the presence of the proposed controller and simulation results are presented.

CloudCL: Single-Paradigm Distributed Heterogeneous Computing for Cloud Infrastructures

The ever-growing demand for compute resources has reached a wide range of application domains, and with that has created a larger audience for compute-intensive tasks. In this paper, we present the CloudCL framework, which empowers users to run compute-intensive tasks without having to face the total cost of ownership of operating an extensive high-performance compute infrastructure. CloudCL enables developers to tap the ubiquitous availability of cloud-based heterogeneous resources using a single-paradigm compute framework, without having to consider dynamic resource management and inter-node communication. In an extensive performance evaluation, we demonstrate the feasibility of the framework, yielding close-to-linear scale-out capabilities for certain workloads.

A method of determining maximum transmission rate in wireless sensor network

The lifetime of wireless sensor networks (WSNs) decreases with the increasing payload, such that extending the network lifetime has become a hot topic. This paper proposes a distance-based transmission rate sexlection algorithm, maximum emission rate (MER) determination, to select the pairwise maximum effective transmission rate in a given WSN environment. The proposed work is founded on two observations, one is the logarithmic relationship of internode communication distance to data transmission rate, the other is the linear relationship of receiving current to data transmission rate. The proposed work helps to reduce data transmission duration, and finally to increase the network lifetime. Simulation results show that the application of MER in existing network protocols extends 300% plus network lifetime while ensuring communication success rate.

UNION: A Trust Model Distinguishing Intentional and Unintentional Misbehavior in Inter-UAV Communication

Ensuring the desired level of security is an important issue in all communicating systems, and it becomes more challenging in wireless environments. Flying Ad Hoc Networks (FANETs) are an emerging type of mobile network that is built using energy-restricted devices. Hence, the communications interface used and that computation complexity are additional factors to consider when designing secure protocols for these networks. In the literature, various solutions have been proposed to ensure secure and reliable internode communications, and these FANET nodes are known as Unmanned Aerial Vehicles (UAVs). In general, these UAVs are often detected as malicious due to an unintentional misbehavior related to the physical features of the UAVs, the communication mediums, or the network interface. In this paper, we propose a new context-aware trust-based solution to distinguish between intentional and unintentional UAV misbehavior. The main goal is to minimize the generated error ratio while meeting the desired security levels. Our proposal simultaneously establishes the inter-UAV trust and estimates the current context in terms of UAV energy, mobility pattern, and enqueued packets, in order to ensure full context awareness in the overall honesty evaluation. In addition, based on computed trust and context metrics, we also propose a new inter-UAV packet delivery strategy. Simulations conducted using NS2.35 evidence the efficiency of our proposal, called UNION, at ensuring high detection ratios > 87% and high accuracy with reduced end-to-end delay, clearly outperforming previous proposals known as RPM, T-CLAIDS, and CATrust.

Distributed Partition Detection With Dynamic Replication Management in a DHT-Based MANET

Curtailing the information loss due to network partitioning is imperative for the reliable communication in a distributed hash table (DHT)-based routing protocol for mobile ad hoc networks (MANETs). Both the limited transmission range and mobility of nodes cause recurrent network partitioning in MANETs that leads to an inaccessibility of the nodes’ mapping information, loss of the LID space, and partitioning of the DHT structure. To address the above-mentioned issues, this paper proposes a protocol that employs pre-partitioning measures, such as the critical link/node detection, the replication of the nodes’ mapping information, and so on, to augment the inter-node communication, but also reflects substantial improvement in the lookup delay. The proposed protocol effectively exploits the local neighbor information in a 3-D logical structure to identify/detect the critical nodes by employing a localized distributed algorithm. Simulation results show the significance of the proposed protocol in terms of end-to-end lookup delay, routing overhead, and success ratio.

Data-Reserved Periodic Diffusion LMS With Low Communication Cost Over Networks

In this paper, we analyze diffusion strategies in which all nodes attempt to estimate a common vector parameter for achieving distributed estimation in adaptive networks. Under diffusion strategies, each node essentially needs to share processed data with predefined neighbors. Although the use of internode communication has contributed significantly to improving convergence performance based on diffusion, such communications consume a huge quantity of power in data transmission. In developing low-power consumption diffusion strategies, it is very important to reduce the communication cost without significant degradation of convergence performance. For that purpose, we propose a data-reserved periodic diffusion least-mean-squares (LMS) algorithm in which each node updates and transmits an estimate periodically while reserving its measurement data even during non-update time. By applying these reserved data in an adaptation step at update time, the proposed algorithm mitigates the decline in convergence speed incurred by most conventional periodic schemes. For a period $p$ , the total cost of communication is reduced to a factor of $1/p$ relative to the conventional adapt-then-combine (ATC) diffusion LMS algorithm. The loss of combination steps in this process leads naturally to a slight increase in the steady-state error as the period $p$ increases, as is theoretically confirmed through mathematical analysis. We also prove an interesting property of the proposed algorithm, namely, that it suffers less degradation of the steady-state error than the conventional diffusion in a noisy communication environment. Experimental results show that the proposed algorithm outperforms related conventional algorithms and, in particular, outperforms ATC diffusion LMS over a network with noisy links.

A hybrid parallel Delaunay image-to-mesh conversion algorithm scalable on distributed-memory clusters

In this paper, we present a scalable three dimensional parallel Delaunay image-to-mesh conversion algorithm. A nested master–worker model is used to simultaneously explore process- and thread-level parallelization. The mesh generation includes two stages: coarse and fine meshing. First, a coarse mesh is constructed in parallel by the threads of the master process. Then the coarse mesh is partitioned. Finally, the fine mesh refinement procedure is executed until all the elements in the mesh satisfy the quality and fidelity criteria. The communication and computation are separated during the fine mesh refinement procedure. The master thread of each process that initializes the MPI environment is in charge of the inter-node MPI communication for data (submesh) movement while the worker threads of each process are responsible for the local mesh refinement within the node. We conducted a set of experiments to test the performance of the algorithm on distributed memory clusters and observed that the granularity of coarse level data decomposition, which affects the coarse level concurrency, has a significant influence on the performance of the algorithm. With the proper value of granularity, the algorithm is scalable to 45 distributed memory compute nodes (900 cores).

Running large‐scale CFD applications on Intel‐KNL–based clusters

SummaryIntel's latest Xeon Phi processor, Knights Landing (KNL), has the potential to provide over 2.6 TFLOPS. However, to obtain maximum performance on the KNL, significant refactoring and optimization of application codes are still required to exploit key architectural innovations that KNL features—wide vector units, many‐core node design, and deep memory hierarchy. The experience and insights gained in porting and running FEFLO (a typical edge‐based finite element code for the solution of compressible and incompressible flows) on the KNL platform are described in this paper. In particular, optimizations used to extract on‐node parallelism via vectorization and multithreading and improve internode communication are considered. These optimizations resulted in a 2.3× performance gain on a 16 node runs of FEFLO, with the potential for larger performance gains as the code is scaled beyond 16 nodes. The impact of the different configurations of KNL's on‐package MCDRAM (Multi‐Channel DRAM) memory on FEFLO's performance is also explored. Finally, the performance of the optimized versions of FEFLO for KNL and Haswell (Intel Xeon) is compared.

Power Sharing in Angle Droop Controlled Microgrids

Component mismatches and parameters drifts drastically affect stability and long-term operation of droop-controlled inverter-based microgrids. This paper analyzes and illustrates the impact of design variations and parameter drifts between angle droop controlled inverter-interfaced sources in a microgrid. It is shown that microgrid stability is very sensitive to parameter drifts, especially in frequency. A coordination control scheme that uses internode communications is proposed for improving the stability margin and ensuring the desired power sharing. Conditions for stability are derived and simulation results are presented to validate the performance of the proposal.

Performance Evaluation of Runtime Data Exploration Framework based on In-Situ Particle Based Volume Rendering

We examine the performance of the in-situ data exploration framework based on the in-situ Particle Based Volume Rendering (In-Situ PBVR) on the latest many-core platform. In-Situ PBVR converts extreme scale volume data into small rendering primitive particle data via parallel Monte-Carlo sampling without costly visibility ordering. This feature avoids severe bottlenecks such as limited memory size per node and significant performance gap between computation and inter-node communication. In addition, remote in-situ data exploration is enabled by asynchronous file-based control sequences, which transfer the small particle data to client PCs, generate view-independent volume rendering images on client PCs, and change visualization parameters at runtime. In-Situ PBVR shows excellent strong scaling with low memory usage up to ~100k cores on the Oakforest-PACS, which consists of 8,208 Intel Xeon Phi7250 (Knights Landing) processors. This performance is compatible with the remote in-situ data exploration capability.

Communication-Efficient Decentralized Sparse Bayesian Learning of Joint Sparse Signals

We consider the problem of decentralized estimation of multiple joint sparse vectors by a network of nodes from locally acquired noisy and underdetermined linear measurements, when the cost of communication between the nodes is at a premium. We propose an iterative, decentralized Bayesian algorithm called fusion-based distributed sparse Bayesian learning (FB-DSBL) in which the nodes collaborate by exchanging highly compressed messages to learn a common joint sparsity inducing signal prior. The learnt signal prior is subsequently used by each node to compute the maximum a posteriori probability estimate of its respective sparse vector. Since the internode communication cost is expensive, the size of the messages exchanged between nodes is reduced substantially by exchanging only those local signal prior parameters which are associated with the nonzero support detected via multiple composite log-likelihood ratio tests. The average message size is empirically shown to be proportional to the information rate of the unknown vectors. The proposed sparse Bayesian learning (SBL)-based distributed algorithm allows nodes to exploit the underlying joint sparsity of the signals. In turn, this enables the nodes to recover sparse vectors with significantly lower number of measurements compared to the standalone SBL algorithm. The proposed algorithm is interpreted as a degenerate case of a distributed consensus-based stochastic approximation algorithm for finding a fixed point of a function, and its generalized version with Robbins–Monro-type iterations is also developed. Using Monte Carlo simulations, we demonstrate that the proposed FB-DSBL has superior mean squared error and support recovery performance compared to the existing decentralized algorithms with similar or higher communication complexity.

Scalable Single Source Shortest Path Algorithms for Massively Parallel Systems

We consider the single-source shortest path (SSSP) problem: given an undirected graph with integer edge weights and a source vertex $v$ , find the shortest paths from $v$ to all other vertices. In this paper, we introduce a novel parallel algorithm, derived from the Bellman-Ford and Delta-stepping algorithms. We employ various pruning techniques, such as edge classification and direction-optimization, to dramatically reduce inter-node communication traffic, and we propose load balancing strategies to handle higher-degree vertices. These techniques are particularly effective on power-law graphs, as demonstrated by our extensive performance analysis. In the largest tested configuration, an R-MAT graph with $2^{38}$ vertices and $2^{42}$ edges on 32,768 Blue Gene/Q nodes, we have achieved a processing rate of three Trillion Edges Per Second (TTEPS), a four orders of magnitude improvement over the best published results.

Price-based data delivery framework for dynamic and pervasive IoT

Recent ongoing research efforts are taking advantage of the emerging technologies which realize the Internet of Things (IoT) paradigm. However, the realization of an IoT framework is still facing many challenges in terms of QoS, resource management and pricing, in addition to the most pressing challenge which is attributed to the integration of the heterogeneous nodes and devices that compete to efficiently route their data. Routing data and inter-nodal communication in IoT has not been yet well investigated; primarily due to the absolute reliance on the Internet as a backhaul. Nevertheless, spatially linked entities in the IoT, and those which most often interact, would pose a significant overhead of communication if all intermediate data packets need to be routed over such distant/dynamic topology. Consequently, we propose an Adaptive Routing Approach (ARA) that selectively launches routes of communication between the heterogeneous IoT nodes. Since nodes in the IoT belong to different owners, we also introduce a pricing mechanism to cater for the exchange of monetary costs by intermediate nodes to utilize their relaying resources. In this paper, the ARA routing approach is described and a use case in addition to theoretical analysis to demonstrate the utility and practicality of ARA in the heterogeneous IoT as it scales is investigated. Moreover, extensive simulation results are performed to outline the efficiency of the ARA approach under IoT-specific setups.

OUTAGE ANALYSIS OF LOW-LATENCY COOPERATIVE WIRELESS NETWORKS WITH THRESHOLD-BASED PROTOCOL OVER COMPOSITE FADING

This paper studies a low-latency decode-and-forward cooperative wireless network subject to composite fading. Assuming temporally correlated channel between cooperating nodes and maximal ratio combining at the destination, outage probability (OP) performance is investigated and novel OP expressions are derived when nodes apply a threshold-based protocol for internode communication. The effects of network dimension, multipath fading and shadowing severity parameters, correlation coefficients and average uplink and internode signal-to-noise ratios on OP are discussed.

Decentralized Joint-Sparse Signal Recovery: A Sparse Bayesian Learning Approach

This work proposes a decentralized, iterative, Bayesian algorithm called CB-DSBL for in-network estimation of multiple jointly sparse vectors by a network of nodes, using noisy and underdetermined linear measurements. The proposed algorithm exploits the network wide joint sparsity of the un- known sparse vectors to recover them from significantly fewer number of local measurements compared to standalone sparse signal recovery schemes. To reduce the amount of inter-node communication and the associated overheads, the nodes exchange messages with only a small subset of their single hop neighbors. Under this communication scheme, we separately analyze the convergence of the underlying Alternating Directions Method of Multipliers (ADMM) iterations used in our proposed algorithm and establish its linear convergence rate. The findings from the convergence analysis of decentralized ADMM are used to accelerate the convergence of the proposed CB-DSBL algorithm. Using Monte Carlo simulations, we demonstrate the superior signal reconstruction as well as support recovery performance of our proposed algorithm compared to existing decentralized algorithms: DRL-1, DCOMP and DCSP.

3D Kirchhoff depth migration algorithm: A new scalable approach for parallelization on multicore CPU based cluster

In this article, a new scalable 3D Kirchhoff depth migration algorithm is presented on state of the art multicore CPU based cluster. Parallelization of 3D Kirchhoff depth migration is challenging due to its high demand of compute time, memory, storage and I/O along with the need of their effective management. The most resource intensive modules of the algorithm are traveltime calculations and migration summation which exhibit an inherent trade off between compute time and other resources. The parallelization strategy of the algorithm largely depends on the storage of calculated traveltimes and its feeding mechanism to the migration process. The presented work is an extension of our previous work, wherein a 3D Kirchhoff depth migration application for multicore CPU based parallel system had been developed. Recently, we have worked on improving parallel performance of this application by re-designing the parallelization approach. The new algorithm is capable to efficiently migrate both prestack and poststack 3D data. It exhibits flexibility for migrating large number of traces within the available node memory and with minimal requirement of storage, I/O and inter-node communication. The resultant application is tested using 3D Overthrust data on PARAM Yuva II, which is a Xeon E5-2670 based multicore CPU cluster with 16 cores/node and 64GB shared memory. Parallel performance of the algorithm is studied using different numerical experiments and the scalability results show striking improvement over its previous version. An impressive 49.05X speedup with 76.64% efficiency is achieved for 3D prestack data and 32.00X speedup with 50.00% efficiency for 3D poststack data, using 64 nodes. The results also demonstrate the effectiveness and robustness of the improved algorithm with high scalability and efficiency on a multicore CPU cluster.