Fat-tree Network Research Articles

3D DFT by block tensor-matrix multiplication via a modified Cannon's algorithm: Implementation and scaling on distributed-memory clusters with fat tree networks

A known scalability bottleneck of the parallel 3D FFT is its use of all-to-all communications. Here, we present S3DFT, a library that circumvents this by using point-to-point communication – albeit at a higher arithmetic complexity. This approach exploits three variants of Cannon's algorithm with adaptations for block tensor-matrix multiplications. We demonstrate S3DFT's efficient use of hardware resources, and its scaling using up to 16,464 cores of the JUWELS Cluster. However, in a comparison with well-established 3D FFT libraries, its parallel efficiency and performance were found to fall behind. A detailed analysis identifies the cause in two of its component algorithms, which scale poorly owing to how their communication patterns are mapped in subsets of the fat tree topology. This result exposes a potential drawback of running block-wise parallel algorithms on systems with fat tree networks caused by increased communication latencies along specific directions of the mesh of processing elements.

Load balancing and topology dynamic adjustment strategy for power information system network: a deep reinforcement learning-based approach

As power information systems play an increasingly critical role in modern society, higher requirements are placed on the performance and reliability of their network infrastructure. In order to cope with the growing data traffic and network attack threats in the power information system, we select the power information system data center network as the research object and design an overall system solution based on software defined network, including the application layer, control layer and infrastructure layer. A typical fat tree network topology is simulated and analyzed. We define the load balancing and network topology dynamic adjustment problem as a Markov decision process, and design a data flow path acquisition method based on breadth-first search to construct the action space of each host. Then, a deep reinforcement learning algorithm based on deep Q-network, priority experience replay and target network is introduced to provide solutions for optimizing the performance of power information systems and responding to network attacks. Simulation results show that the proposed method is better than the traditional equal-cost multi-path algorithm in terms of average bandwidth utilization, average jitter and average packet loss, and can reduce the probability of network nodes being attacked by more than 11%.

Modular Control Plane Verification via Temporal Invariants

Monolithic control plane verification cannot scale to hyperscale network architectures with tens of thousands of nodes, heterogeneous network policies and thousands of network changes a day. Instead, modular verification offers improved scalability, reasoning over diverse behaviors, and robustness following policy updates. We introduce Timepiece, a new modular control plane verification system. While one class of verifiers, starting with Minesweeper, were based on analysis of stable paths, we show that such models, when deployed naïvely for modular verification, are unsound. To rectify the situation, we adopt a routing model based around a logical notion of time and develop a sound, expressive, and scalable verification engine. Our system requires that a user specifies interfaces between module components. We develop methods for defining these interfaces using predicates inspired by temporal logic, and show how to use those interfaces to verify a range of network-wide properties such as reachability or access control. Verifying a prefix-filtering policy using a non-modular verification engine times out on an 80-node fattree network after 2 hours. However, Timepiece verifies a 2,000-node fattree in 2.37 minutes on a 96-core virtual machine. Modular verification of individual routers is embarrassingly parallel and completes in seconds, which allows verification to scale beyond non-modular engines, while still allowing the full power of SMT-based symbolic reasoning.

Parallel Scalability of Three-Level FROSch Preconditioners to 220000 Cores using the Theta Supercomputer

The parallel performance of the three-level fast and robust overlapping Schwarz (FROSch) preconditioners is investigated for linear elasticity. The FROSch framework is part of the Trilinos software library and contains a parallel implementation of different preconditioners with energy minimizing coarse spaces of generalized Dryja--Smith--Widlund type. The three-level extension is constructed by a recursive application of the FROSch preconditioner to the coarse problem. In this paper, the additional steps in the implementation in order to apply the FROSch preconditioner recursively are described in detail. Furthermore, it is shown that no explicit geometric information is needed in the recursive application of the preconditioner. In particular, the rigid body modes, including the rotations, can be interpolated on the coarse level without additional geometric information. Parallel results for a three-dimensional linear elasticity problem obtained on the Theta supercomputer (Argonne Leadership Computing Facility, Argonne, IL) using up to 220 000 cores are discussed and compared to results obtained on the SuperMUC-NG supercomputer (Leibniz Supercomputing Centre, Garching, Germany). Notably, it can be observed that a hierarchical communication operation in FROSch related to the coarse operator starts to dominate the computing time on Theta, which has a dragonfly interconnect, for 100 000 message passing interface (MPI) ranks or more. The same operation, however, scales well and stays within the order of a second in all experiments performed on SuperMUC-NG, which uses a fat tree network. Using hybrid MPI/OpenMP parallelization, the onset of the MPI communication problem on Theta can be delayed. Further analysis of the performance of FROSch on large supercomputers with dragonfly interconnects will be necessary.

Deep Reinforcement Learning-Based Network Routing Technology for Data Recovery in Exa-Scale Cloud Distributed Clustering Systems

Research has been conducted to efficiently transfer blocks and reduce network costs when decoding and recovering data from an erasure coding-based distributed file system. Technologies using software-defined network (SDN) controllers can collect and more efficiently manage network data. However, the bandwidth depends dynamically on the number of data transmitted on the network, and the data transfer time is inefficient owing to the longer latency of existing routing paths when nodes and switches fail. We propose deep Q-network erasure coding (DQN-EC) to solve routing problems by converging erasure coding with DQN to learn dynamically changing network elements. Using the SDN controller, DQN-EC collects the status, number, and block size of nodes possessing stored blocks during erasure coding. The fat-tree network topology used for experimental evaluation collects elements of typical network packets, the bandwidth of the nodes and switches, and other information. The data collected undergo deep reinforcement learning to avoid node and switch failures and provide optimized routing paths by selecting switches that efficiently conduct block transfers. DQN-EC achieves a 2.5-times-faster block transmission time and 0.4-times-higher network throughput than open shortest path first (OSPF) routing algorithms. The bottleneck bandwidth and transmission link cost can be reduced, improving the recovery time approximately twofold.

A Software Defined Network Scheme for Intra Datacenter Network Based on Fat-Tree Topology

Nowadays datacenters face a big challenge, because the development of cloud services requests a large amount of capital input, high maintenance skills and cost. The major disadvantages of current datacenter network include the high cost of equipment and maintenance, QoS, failure recovery time, network virtualization etc. In this paper, we proposed a software defined datacenter network. The programmable controller is responsible for the management of the traffic scheduling including topology discovery, routing and Quality of Service. The SDN is based on Fat-tree network architecture and max-min fairness. A Genetic Algorithm optimized Radial Basis Function neural network is used to compute the service weight for maximizing the utilization of network resource. An experiment of the network bearing different kinds of services based on Openflow is described. This well-designed fat-tree network has high efficiency in traffic distribution, and has integrated the traditional network architecture and new SDN technology.

A network-aware and power-efficient virtual machine placement scheme in cloud datacenters based on chemical reaction optimization

In the present article, we propose a virtual machine placement (VMP) algorithm for reducing power consumption in heterogeneous cloud data centers. We propose a novel model for the estimation of power consumption of datacenter’s network. The proposed model is employed to estimate power consumption of a Fat-Tree network. It calculates the traffic of each network layer and uses the results to estimate the average power consumption of each switch in the network, which is used for network power calculation. Further, we employ the chemical reaction optimization (CRO) algorithm as a meta-heuristic algorithm to obtain a power-efficient mapping of virtual machines (VMs) to physical machines (PMs). Moreover, two kinds of solution encoding schemes, namely permutation-based and grouping-based encoding schemes, were utilized for representing individuals in CRO. For each encoding scheme, we designed proper operators required by the CRO for manipulating the molecules in search of more optimal solution candidates. Additionally, we modeled VMs with east–west and north–south communications, and PMs with constrained CPU, memory, and bandwidth capacity. Our network power model is integrated into the CRO algorithms to enable the estimation of both PMs and network power consumption. We compared our proposed methods with a number of similar methods. The evaluation results indicate that the proposed methods perform well and the CRO algorithm with the grouping-based encoding outperforms the rest of the methods in terms of power consumption. The evaluation results also show the significance of network power consumption.

Enhancing Robustness of Per-Packet Load-Balancing for Fat-Tree

Fat-tree networks have many equal-cost redundant paths between two hosts. To achieve low flow completion time and high network utilization in fat-tree, there have been many efforts to exploit topological symmetry. For example, packet scatter schemes, which spray packets across all equal-cost paths relying on topological symmetry, work well when there is no failure in networks. However, when symmetry of a network is disturbed due to a network failure, packet scatter schemes may suffer massive packet reordering. In this paper, we propose a new load balancing scheme named LBSP (Load Balancing based on Symmetric Path groups) for fat-trees. LBSP partitions equal-cost paths into equal sized path groups and assigns a path group to each flow so that packets of a flow are forwarded across paths within the selected path group. When a link failure occurs, the flows affected by the failure are assigned an alternative path group which does not contain the failed link. Consequently, packets in one flow can still experience almost the same queueing delay. Simulation results show that LBSP is more robust to network failures compared to the original packet scatter scheme. We also suggest a solution to the queue length differentials between path groups.

Segment Switching: A New Switching Strategy for Optical HPC Networks

Photonics are becoming realistic technologies for implementing interconnection networks in near future Exascale supercomputer systems. Photonics present key features to design high-performance and scalable supercomputer networks, such as higher bandwidth and lower latencies than their electronic supercomputer networks counterparts. Some research work is focused on conventional network topologies built with photonic technologies, with the aim of taking advantage of photonic characteristics. Nevertheless, these approaches fail in that they keep low the network utilization. We looked into this downside and we found that circuit switching was the main performance limitation. In this article we propose a new switching mechanism, called Segment Switching , to address this constraint and improve the network utilization. Segment Switching splits the circuit in segments of the whole path, and uses buffering on selected nodes on the network. Experimental results show that the devised approach significantly outperforms photonic circuit switching in conventional torus and fat tree networks by 70% and 90%, respectively.

Toward lower-diameter large-scale HPC and data center networks with co-packaged optics

We investigate the advantages of using co-packaged optics for building low-diameter, large-scale high-performance computing (HPC) and data center networks. The increased escape bandwidth offered by co-packaged optics can enable high-radix switch implementations of more than 150 switch ports, which can be combined with data rates of up to 400 Gb/s per port. From the network architecture perspective, the key benefits of using co-packaged optics in future fat-tree networks include (a) the ability to implement large-scale topologies of > --> 11 , 000 end points by eliminating the need for a third switching layer and (b) the ability to provide up to 4 × higher bisection bandwidth compared to existing solutions, reducing at the same time the number of required switch application-specific integrated circuits by > --> 80 % . From the network operation perspective, both reduced energy consumption and lower packet delays can be achieved since fewer hops are required; i.e., packets need to traverse fewer serializer/deserializer lanes and fewer switch buffers, which reduces the probability of contending with other packets and improves the tolerance of network congestion. The performance of the proposed architecture is evaluated via discrete-event simulations for a wide range of representative HPC synthetic-traffic cases that include both hotspot and non-hotspot scenarios. The simulation results suggest that co-packaged optics form a promising solution to keep up with bandwidth scaling in future networks, while the reduced number of switching layers can lead to significant mean packet delay improvements that start from 30% and reach up to 74% for high-load conditions.

Load balancing using openday light SDN controller: Case study

Traditional networking architectures have many limitations that need to be overcome to meet modern IT requirements. To overcome these limitations; Software Defined Networking (SDN) is taking place as the new networking approach. As traditional networking uses static switches, resource utilization is poor. There are packet loss and delay during switch breakdown. This paper proposes an implementation of a load balancing algorithm for an SDN based network to overcome the stated issues. To test the algorithm, a network is emulated using the Mininet and OpenDaylight platform (ODL) is used as an SDN controller. Python coding language is used to create fat-tree network topology and to write a load balancing algorithm. Finally, iPerf and Wireshark is used to test network performance. The network was tested before and after running the load balancing algorithm. The testing focused on some of the Quality of Service (QoS) parameters such as bandwidth and transfer rate in the fat-tree network. The algorithm increased bandwidth with at least more than 50%, and improved network utilization

Data privacy‐based coordinated placement method of workflows and data

With the rapid development of data acquisition technology, many industries data already have the characteristics of big data and cloud technology has provided strong support for the storage and complex calculations of these massive data. The meteorological department established the cloud data centre based on the existing storage and computing resources and re-arranged the historical data to reduce the historical data access time of applications. However, the placement of each workflow and input data also affects the average data access time, which in turn affects the computing efficiency of the cloud data centre. At the same time, because of the collaborative processing of multiple nodes, the resource utilisation of cloud data centre has also been paid more and more attention. In addition, with the increase of data security requirements, some privacy conflict data should avoid being placed on the same or neighbouring nodes. In response to this challenge, based on the fat-tree network topology, this study proposes a data privacy protection-based collaborative placement strategy of workflow and data to jointly optimise the average data access time, the average resource utilisation, and the data conflict degree. Finally, a large number of experimental evaluations and comparative analyses verify the efficiency of the proposed method.

Towards Network-Aware Service Composition in the Cloud

Composing several API-defined services into one composite service per user requirements has become an important service creation approach in the cloud-enabled API economy. Various service selection approaches in support of service composition on demand have been proposed. They usually assume that networking resources are over-provisioned and their usage needs not be considered when making quality-aware service composition decisions. In practice, these approaches often lead to wasteful network resource consumption and impractical end-to-end QoS optimality for cloud-based services. This paper proposes a network-aware cloud service composition approach, named NetMIP, with comparative experimental evaluations for the clouds that adopt the widely deployed fat-tree network topology. By formalizing the service composition goal as a multi-objective constraint optimization problem, we have validated the proposed approach can be used to effectively reduce network resource consumption and deliver QoS optimality while satisfying the end-to-end QoS constraints for the candidate composite services in the cloud. The comparative experimental evaluations are done via a credible cloud infrastructure simulation system, named WebCloudSim. Extensive evaluation results show that NetMIP outperforms several representative cloud service composition approaches in terms of network resource consumption, QoS optimality, and computation time under various service selection workloads and fat-tree network topology settings.

Towards an efficient combination of adaptive routing and queuing schemes in Fat-Tree topologies

The interconnection network is a key element in High-Performance Computing (HPC) and Datacenter (DC) systems whose performance depends on several design parameters, such as the topology, the switch architecture, and the routing algorithm. Among the most common topologies in HPC systems, the Fat-Tree offers several shortest-path routes between any pair of end-nodes, which allows multi-path routing schemes to balance traffic flows among the available links, thus reducing congestion probability. However, traffic balance cannot solve by itself some congestion situations that may still degrade network performance. Another approach to reduce congestion is queue-based flow separation, but our previous work shows that multi-path routing may spread congested flows across several queues, thus being counterproductive. In this paper, we propose a set of restrictions to improve alternative routes selection for multi-path routing algorithms in Fat-Tree networks, so that they can be positively combined with queuing schemes.

A Scalable, High-Performance, and Fault-Tolerant Network Architecture for Distributed Machine Learning

In large-scale distributed machine learning (DML), the network performance between machines significantly impacts the speed of iterative training. In this paper we propose BML, a scalable, high-performance and fault-tolerant DML network architecture on top of Ethernet and commodity devices. BML builds on BCube topology, and runs a fully-distributed gradient synchronization algorithm. Compared to a Fat-Tree network with the same size, a BML network is expected to take much less time for gradient synchronization, for both low theoretical synchronization time and its benefit to RDMA transport. With server/link failures, the performance of BML degrades in a graceful way. Experiments of MNIST and VGG-19 benchmarks on a testbed with 9 dual-GPU servers show that, BML reduces the job completion time of DML training by up to 56.4% compared with Fat-Tree running state-of-the-art gradient synchronization algorithm.

The high-speed networks of the Summit and Sierra supercomputers

Oak Ridge National Laboratory's Summit supercomputer and Lawrence Livermore National Laboratory's Sierra supercomputer utilize InfiniBand interconnect in a Fat-tree network topology, interconnecting all compute nodes, storage nodes, administration, and management nodes into one linearly scalable network. These networks are based on Mellanox 100-Gb/s EDR InfiniBand ConnectX-5 adapters and Switch-IB2 switches, with compute-rack packaging and cooling contributions from IBM. These devices support in-network computing acceleration engines such as Mellanox Scalable Hierarchical Aggregation and Reduction Protocol, graphics processor unit (GPU) Direct RDMA, advanced adaptive routing, Quality of Service, and other network and application acceleration. The overall IBM Spectrum Message Passing Interface (MPI) messaging software stack implements Open MPI, and was a collaboration between IBM, Mellanox, and NVIDIA to optimize direct communication between endpoints, whether compute nodes (with IBM POWER CPUs, NVIDIA GPUs, and flash memory devices), or POWER-hosted storage nodes. The Fat-tree network can isolate traffic among the compute partitions and to/from the storage subsystem, providing more predictable application performance. In addition, the high level of redundancy of this network and its reconfiguration capability ensures reliable high performance even after network component failures. This article details the hardware and software architecture and performance of the networks and describes a number of the high-performance computing (HPC) enhancements engineered into this generation of InfiniBand.

Evaluation of Firewall and Load balance in Fat-Tree Topology Based on Floodlight Controller

Today it has become important to reconfigure the networks in to new form to be more manageable, scalable, dynamic and programmable. The networks recently are so inflexible and failing to deal with the required changes for the Information Technology. Software Defined Networking (SDN) is a modern paradigm that focused to change the main idea of current network infrastructure (traditional network) by breaking the chain between the data forwarding and the control planes to introduce flexible programmability network. This paper makes comparison between the performance of traditional fat-tree network and SDN fat-tree network, which found that average Round Tripe Time (RTT) in SDN fat-tree topology will decrease by 8.96% than traditional fat-tree topology. Then shows the basic operation of OpenFlow protocol that can be applied on fat-tree topology by using SDN technology and how that can be effect on the performance of network and make it more flexible to enable the SDN module applications, like load balancer and firewall for optimizing the SDN network. In this paper the physical switches are replaced by software switches in a virtual network environment and display the SDN structure in GUI, also Floodlight controller is chosen to use as the network operating system for SDN network.

Performance drop at executing communication-intensive parallel algorithms

This work summarizes the results of a set of executions completed on three fat-tree network supercomputers: Stampede at TACC (USA), Helios at IFERC (Japan) and Eagle at PSNC (Poland). Three MPI-based, communication-intensive scientific applications compiled for CPUs have been executed under weak-scaling tests: the molecular dynamics solver LAMMPS; the finite element-based mini-kernel miniFE of NERSC (USA); and the three-dimensional fast Fourier transform mini-kernel bigFFT of LLNL (USA). The design of the experiments focuses on the sensitivity of the applications to rather different patterns of task location, to assess the impact on the cluster performance. The accomplished weak-scaling tests stress the effect of the MPI-based application mappings (concentrated vs. distributed patterns of MPI tasks over the nodes) on the cluster. Results reveal that highly distributed task patterns may imply a much larger execution time in scale, when several hundreds or thousands of MPI tasks are involved in the experiments. Such a characterization serves users to carry out further, more efficient executions. Also researchers may use these experiments to improve their scalability simulators. In addition, these results are useful from the clusters administration standpoint since tasks mapping has an impact on the cluster throughput.

OPS: A Fairness Link Allocation Based on SDN in Datacenter Networks

Nowadays, redundant topologies with multiple forwarding paths between original-destination (OD) pairs have been proposed for datacentre networks, such as fat-tree topology. To achieve resultant fairness (fair utilisation) in this topology, previous algorithms were based on traditional shortest path algorithms or equal-cost multiple paths algorithms (ECMP). However, these algorithms are facing with take up a lot of resources of the SDN controller and may cause a large amount of loss of resultant fairness for the network that has different configurations of link capacity. In this paper, we present out port selection (OPS) mechanisms, including modular out port selection (MOPS) and random out port selection (ROPS), novel methods to improve resultant fairness of fat-tree topology SDN-based datacentre network. Compare with static load-balance methods, our MOPS and ROPS can reduce 65% and 33% loss of fairness, respectively. Further, combining with elephant flows detecting and handling mechanisms, MOPSE and ROPSE can dynamically assign better forwarding path for elephant flows.

Path2SL: Leveraging InfiniBand Resources to Reduce Head-of-Line Blocking in Fat Trees

The number of endnodes in high-performance computing and datacenter systems is constantly increasing. Hence, it is crucial to minimize the impact of network congestion to guarantee a suitable network performance. InfiniBand is a prominent interconnect technology that allows implementing efficient topologies and routing algorithms, as well as queuing schemes that reduce the head-of-line (HoL) blocking effect derived from congestion situations. Here, we explain and evaluate thoroughly a queuing scheme called Path2SL that optimizes the use of the InfiniBand Virtual Lanes to reduce the HoL blocking in fat-tree network topologies.