Top Of Rack Research Articles

Congestion control analysis of optical packet switch for optical data center applications

Abstract Optical data centers serve as the backbone of modern networking, facilitating seamless connectivity for users across the globe. The connection between users and the optical data centers is established through various network topologies, which play a critical role in determining the traffic characteristics. The design and implementation of these network topologies, along with the assortment of applications hosted on optical data centers, significantly influence the flow of data within the network. The advent of cloud computing has further revolutionized optical data center operations, leading to the coexistence of a wide range of applications on different optical data center switches. As a result, the traffic characteristics observed on each optical data center switch vary significantly. This diversity in traffic patterns necessitates a comprehensive understanding of how data arrival is managed and handled by the networking infrastructure. In this paper, we explore the concept of a random traffic model for data arrival on Top of Rack (ToR) switches, which represent a crucial component of optical data center networking. In the modeling, small world model is considered. The effect of buffering and packet priorities is observed on traffic shaping. Finally, to evaluate the effectiveness of the traffic shaping techniques, we measure the packet loss performance of ToR switches and found to be as low as 10−4 even at the higher loads. Blocking performance provides valuable insights into how effectively the optical data center network manages incoming data and avoids congestion or bottlenecks.

Shufflecast: An Optical, Data-Rate Agnostic, and Low-Power Multicast Architecture for Next-Generation Compute Clusters

An optical circuit-switched network core has the potential to overcome the inherent challenges of a conventional electrical packet-switched core of today’s compute clusters. As optical circuit switches (OCS) directly handle the photon beams without any optical-electrical-optical (O/E/O) conversion and packet processing, OCS-based network cores have the following desirable properties: a) agnostic to data-rate, b) negligible/zero power consumption, c) no need of transceivers, d) negligible forwarding latency, and e) no need for frequent upgrade. Unfortunately, OCS can only provide point-to-point (unicast) circuits. They do not have built-in support for one-to-many (multicast) communication, yet multicast is fundamental to a plethora of data-intensive applications running on compute clusters nowadays. In this paper, we propose Shufflecast, a novel optical network architecture for next-generation compute clusters that can support high-performance multicast satisfying all the properties of an OCS-based network core. Shufflecast leverages small fanout, inexpensive, passive optical splitters to connect the Top-of-rack (ToR) switch ports, ensuring data-rate agnostic, low-power, physical-layer multicast. We thoroughly analyze Shufflecast’s highly scalable data plane, light-weight control plane, and graceful failure handling. Further, we implement a complete prototype of Shufflecast in our testbed and extensively evaluate the network. Shufflecast is more power-efficient than the state-of-the-art multicast mechanisms. Also, Shufflecast is more cost-efficient than a conventional packet-switched network. By adding Shufflecast alongside an OCS-based unicast network, an all-optical network core with the aforementioned desirable properties supporting both unicast and multicast can be realized.

Dynamic Distributed Multi-Path Aided Load Balancing for Optical Data Center Networks

Benefiting from dense connections in data center networks (DCNs), load balancing algorithms are capable of steering traffic into multiple paths for the sake of preventing traffic congestion. However, given each path’s time-varying and asymmetrical traffic state, this may also lead to worse congestion when some paths are overutilised. Especially in the two-tier hybrid optical/electrical DCNs (Hoe-DCNs), the port contentions and large-grained optical packets of the fast optical switch (FOS) require the top-of-rack (TOR) switch to have microsecond-level load balancing capability for microburst traffic. This paper establishes a leaf-spine Hoe-DCN model to illustrate the principal characteristic of dynamic load balancing in TOR switches for the first time. Moreover, we propose the dynamic distributed multi-path (DDMP) load balancing algorithm that relies on dynamic hashing computing for network flow distribution in DCNs, which dynamically adjusts traffic flow distribution at microsecond level according to the inverse ratio of the buffer occupancy. The simulation results show that our proposed algorithm reduces the TOR-to-TOR latency by 15.88% and decreases the packet loss by 22.06% compared to conventional algorithms under regular load conditions, which effectively improves the overall performance of the Hoe-DCNs. Moreover, our proposed algorithm prevents more than 90% packet loss under low load conditions.

Performance trade-offs in reconfigurable networks for HPC

Designing efficient interconnects to support high-bandwidth and low-latency communication is critical toward realizing high performance computing (HPC) and data center (DC) systems in the exascale era. At extreme computing scales, providing the requisite bandwidth through overprovisioning becomes impractical. These challenges have motivated studies exploring reconfigurable network architectures that can adapt to traffic patterns at runtime using optical circuit switching. Despite the plethora of proposed architectures, surprisingly little is known about the relative performances and trade-offs among different reconfigurable network designs. We aim to bridge this gap by tackling two key issues in reconfigurable network design. First, we study how cost, power consumption, network performance, and scalability vary based on optical circuit switch (OCS) placement in the physical topology. Specifically, we consider two classes of reconfigurable architectures: one that places OCSs between top-of-rack (ToR) switches—ToR-reconfigurable networks (TRNs)—and one that places OCSs between pods of racks—pod-reconfigurable networks (PRNs). Second, we tackle the effects of reconfiguration frequency on network performance. Our results, based on network simulations driven by real HPC and DC workloads, show that while TRNs are optimized for low fan-out communication patterns, they are less suited for carrying high fan-out workloads. PRNs exhibit better overall trade-off, capable of performing comparably to a fully non-blocking fat tree for low fan-out workloads, and significantly outperform TRNs for high fan-out communication patterns.

Software-defined optical intra-data center network and access control Strategy

The continuously emerging cloud services provide an unprecedented traffic growth into the large-scale data centers (DCs) globally. In this paper, we introduce an optical DC network (DCN) architecture to organize the servers into computing clusters. Since a high percentage of the total DCΝ traffic is served within a cluster, we assume two distinct networks: the intra-cluster passive optical network that handles the traffic destined to any server of the same cluster and the inter-cluster one to route the traffic to any other cluster. The servers interconnection within the passive optical intra-cluster network causes low power consumption, while the Top-of-Cluster (ToC) switch requires less ports than a relative Top-of-Rack (ToR) one to interconnect the same number of servers within the intra network, reducing even more the total power consumption. In the data plane, the intra- and the inter-cluster networks use separate wavelengths. In the control plane, the software-defined networking (SDN) paradigm is followed. Especially, in each cluster we adopt a cluster controller to coordinate the medium access control (MAC) in both the intra and inter-cluster networks. Unlike other studies that assume electrical connectivity with the controller, we consider that it is performed in the optical domain to guarantee the effective synchronized operation of the control and data planes. In our work, we focus on the intra-cluster network. We propose a synchronous transmission software-defined bandwidth allocation (SD-BA) MAC protocol to fairly coordinate the collisions-free transmission of different quality of service traffic categories in the intra-cluster network, based on the wavelength and time division multiplexing (W&TDM) techniques. The proposed DCN architecture along with the SD-MAC protocol provides scalability and efficiency. Simulations results show that the proposed SD-BA MAC protocol achieves almost 100% bandwidth utilization, while it reaches at high loads 145% higher throughput, 573% lower delay and 233% less dropped packets as compared to the relative DMAC network architecture (Zheng and Sun, Apr. 2020) [24]. Also, the proposed intra-cluster DCN architecture is compared to some other currently leading relative ones in terms of throughput and power consumption and it is proven to be a performance and energy efficient DCN solution.

Balanced Multipath Transport Protocol for Mitigating MPTCP Incast in Data Center Networks

Recent data centers provide dense inter-connectivity between each pair of servers through multiple paths. These data centers offer high aggregate bandwidth and robustness by using multiple paths simultaneously. Multipath TCP (MPTCP) protocol is developed for improving throughput, fairly sharing network link capacity and providing robustness during path failure by utilizing multiple paths over multi-homed data center networks. Running MPTCP protocol for latency-sensitive rack-local short flows with many-to-one communication pattern at the access layer of multi-homed data center networks creates MPTCP incast problem. In this paper, Balanced Multipath TCP (BMPTCP) protocol is proposed to mitigate MPTCP incast problem in multi-homed data center networks. BMPTCP is a window-based congestion control protocol that prevents constant growth of each worker’s subflow congestion window size. BMPTCP computes identical congestion window size for all concurrent subflows by considering bottleneck Top of Rack (ToR) switch buffer size and increasing count of concurrently transmitting workers. This helps BMPTCP to avoid timeout events due to full window loss at ToR switch. Based on current congestion situation at ToR switches, BMPTCP adjust transmission rates of each worker’s subflow so that total amount of data transmitted by all concurrent subflows does not overflow bottleneck ToR switch buffer. Simulation results show that BMPTCP effectively alleviates MPTCP incast. It improves goodput, reduces flow completion time as compared to existing MPTCP and EW-MPTCP protocols.

Two-Aggregator Topology Optimization Using Single Paths in Data Center Networks

This paper focuses on the data aggregation problem to two aggregators in a data center network under the constraint that each source rack must use a single path to each aggregator. We derive bounds on the approximation ratios of two classes of aggregation algorithms- Restricted 1-Round (R1R) and Restricted 2-Round (R2R). We achieve tighter bounds for the problem with additional constraints on the inputs. We propose another strategy using the 2Chain topology for k = 2, where k denotes the maximum degree of each top-of-rack(ToR) switch (number of uplinks in a ToR switch) in the data center and show that the optimal 2Chain cannot have an aggregation time greater than that of the optimal R1R and R2R topologies. For k ≥ 4, we propose a 1-round aggregation algorithm (1R) that uses trees for aggregation. Experimental results illustrate that 1R, R2R and R1R reduce the aggregation time by up to 85, 67 and 67 percent respectively for k = 4 relative to the two-round aggregation algorithm proposed by Wang et al. Moreover, for k = 2, the 2Chain can reduce the aggregation time up to 42 and 24 percent respectively relative to R1R and R2R.

Blocking performance of optically switched data networks

Abstract Large data centre provides the important facilities like storage, computation, communication, etc. Therefore, we must be very careful while designing the infrastructure and networking of the data centre as these factors decides the expenses on implementation and maintenance of the infrastructure. The data servers are connected with Top of Rack (ToR) switches, which are further connected to aggregated/core switches. This paper discusses the blocking performance of the switches used at various levels of optical data networks. However it must be noted that in data centres traffic is not blocked and but it is temporarily stored but if it can’t stored a negative acknowledgement is sent back and packet re-transmission is done. Blocking performance is done to judge the re-transmission of packets. Blocking performance is obtained using mathematical model as well as using Monte Carlo Simulations.

Hybrid Flow Table Installation: Optimizing Remote Placements of Flow Tables on Servers to Enhance PDP Switches for In-Network Computing

Recently, the programmable data plane (PDP) switches have been considered as the key enablers for in-network computing. However, the limited memory resources in them for flow tables might restrict their performance. This work addresses this challenge by studying how to optimize the placements of flow tables in the external memory on multiple servers, and to access them with remote direct memory access (RDMA) for ensuring low latency. Specifically, we consider a data-center network (DCN) that uses PDP switches as top-of-rack (ToR) switches, and propose and optimize the hybrid flow table installation (hFT-INST) on each ToR switch. With hFT-INST, the switch can either store flow tables in its local memory or use RDMA to install and access them remotely in its rack servers. We first design the protocol and operation procedure of hFT-INST. Then, regarding the key problem of hFT-INST, i.e., how to place the flow tables on the external memory on different servers, we take a few practical parameters into account, and formulate a mixed integer linear programming (MILP) model to tackle it. Next, the optimization in the MILP is transformed into a capacitated facility location problem (CFLP) with additional constraints. We further transform it into a ${k}$ -median problem through pre-processing, and design a polynomial-time approximation algorithm to solve the problem. Extensive simulations confirm the performance of our proposed algorithm. We also prototype our design of the hFT-INST, and conduct experiments to demonstrate its feasibility.

ROTOS: A Reconfigurable and Cost-Effective Architecture for High-Performance Optical Data Center Networks

Fast and high capacity optical switching techniques have the potential to enable low latency and high throughput optical data center networks (DCNs) to afford the rapid increasing traffic boosted by multiple applications. Flexibility of the DCN is of key importance to provide adaptive and dynamic bandwidth and capacity to handle the variable traffic patterns of heterogeneous applications. Aiming at improving the network performance and the system flexibility of optical DCNs, we propose and investigate a novel optical DCN architecture named ROTOS based on reconfigurable optical top of rack (ToR) and fast optical switches. In the proposed DCN architecture, the novel optical flexible ToRs employing multiple transceivers (TRXs) and a wavelength selective switch (WSS) are reconfigured by the software-defined networking (SDN) control plane. The bandwidth can be dynamically allocated to the dedicated optical links on-demand according to the desired oversubscription (OV) and intra-/inter-cluster traffic matrix. Numerical investigations of the novel architecture under realistic traffic model indicate that dynamically allocating the TRXs and elastically controlling the WSS, a packet loss below 1E-5 and a server-to-server latency lower than 3 μs can be guaranteed for different traffic patterns at load of 0.4. With respect to the DCN with static interconnections, the average packet loss of ROTOS decreases two orders of magnitude and the average server-to-server latency performance improves by 21.5%. Scalability investigation to a large number of servers shows limited (11%) performance degradation as the network scale from 2560 to 40960 servers. Additionally, the dynamic bandwidth allocation of the DCN is experimentally validated. Network performance results show a packet loss of 0.05 and 5.85 μs end-to-end latency at the load of 0.8. Finally, investigations on the cost and power consumption confirm that the ROTOS DCN architecture has 28.4% lower cost and 35.0% better improvement for power efficiency with respect to the electrical switch based DCNs.

PSNet: Reconfigurable network topology design for accelerating parameter server architecture based distributed machine learning

The bottleneck of Distributed Machine Learning (DML) has shifted from computation to communication. Lots of works have focused on speeding up communication phase from perspective of Parameter Server (PS) architecture, for example resource scheduling. Nonetheless, the performance improvement of these schemes is limited due to the agnostic of the physical topology to the communication pattern of the applications. Concurrently, some articles have also pointed out the impact of topology on DML performance. Besides, our analysis and experimental results also indicate that the general topologies cannot match well with the communication characteristics of DML based on PS architecture. However, to the best of our knowledge, no special topology is tailored for DML. Therefore, in this paper, we propose PSNet, a reconfigurable modular network topology for DML with consideration of the communication characteristics of PS architecture. The main idea of PSNet is that servers are firstly divided into two categories, namely workers configured with high-performance computing capability and parameter servers equipped with multiple Network Interface Cards (NICs). Then Electrical Circuit Switch (ECS) is exploited to connect workers and Top of Rack (ToR) switches for flexibility and reconfigurability in each module. Our theoretical analysis proves that PSNet not only provides high performance for DML tasks, but also achieves high fault tolerance and flexibility. In order to validate the performance of PSNet, we conduct large-scale simulations and small-scale testbed experiments, and the results of experiments demonstrate that PSNet performs 1.89× and 1.92× faster than FatTree for VGG-16 and ResNet50, respectively.

Two-Aggregator Topology Optimization Using Multiple Paths in Data Center Networks

This paper focuses on the data aggregation problem to two aggregators in a data center network under the constraint that each source rack must use a single path to each aggregator. We derive bounds on the approximation ratios of two classes of aggregation algorithms– Restricted 1-Round (R1R) and Restricted 2-Round (R2R). We achieve tighter bounds for the problem with additional constraints on the inputs. We propose another strategy using the 2Chain topology for k = 2, where k denotes the maximum degree of each top-of-rack(ToR) switch (number of uplinks in a ToR switch) in the data center and show that the optimal 2Chain cannot have an aggregation time greater than that of the optimal R1R and R2R topologies. For k $\geq$ ≥ 4, we propose a 1-round aggregation algorithm (1R) that uses trees for aggregation. Experimental results illustrate that 1R, R2R and R1R reduce the aggregation time by up to 85, 67 and 67 percent respectively for k = 4 relative to the two-round aggregation algorithm proposed by Wang et al. Moreover, for k = 2, the 2Chain can reduce the aggregation time up to 42 and 24 percent respectively relative to R1R and R2R.

Timeslot Switching-Based Optical Bypass in Data Center for Intrarack Elephant Flow With an Ultrafast DPDK-Enabled Timeslot Allocator

To alleviate the performance interruption between elephant and mouse flows, a combined optical timeslot switching (OTS) and L2/L3 Top of Rack (ToR) architecture is experimentally demonstrated for the first time in a data center scenario. In this system, elephant flows are identified on the fly by a newly built sFlow-based traffic analyzer and delivered through a separate OTS subsystem. In this way, elephant flows are physically isolated from mouse flows; thus, fewer mutual interruptions are expected. However, previous OTS research has not sufficiently addressed how the timeslot allocator and timeslot allocation should be implemented to adapt to traffic fluctuations, and it is obvious that the current SDN-based timeslot allocation cannot satisfy the OTS system's requirements for time efficiency. Therefore, in this paper, data-plane development kit platform was used to enable an ultrafast online timeslot allocator (TSA) that can process timeslot requests at traffic levels exceeding 1 Tb/s. This demonstration employs two Intel Xeon CPU e5 logical cores for TSA, a private protocol between the TSA and servers, and a novel timeslot allocation algorithm. Using the proposed architecture, mouse flows are seldom interrupted by elephant flows. The power penalty of the OTS system is less than 1.2 dB, which is acceptable for server-to-server interconnections. Furthermore, simulation results show that the system achieves a consistently lower end-to-end latency for all flows than does a traditional ToR-based intrarack data center network (DCN).

FSCOI: A High Fan-Out, Scalable, and Cluster-Based Optical Interconnect for Data Center Networks

In this letter, a high fan-out, scalable, and cluster-based optical interconnect (FSCOI) for data center networks is proposed. FSCOI consists of an optical switch and clusters. Compared to OSA, the improved fan-out of FSCOI is achieved by two steps. The first is a coupler for sending that can multiplex the optical signals from all top-of-rack (ToR) switches within a cluster. The second is a wavelength selective switch for receiving that can talk to all ToR switches within the cluster. FSCOI is based on cluster and is suited to intra-cluster traffic. In addition, the high scalability is achieved by increasing the port usage efficiency of the optical switch. Its performance is evaluated by simulations. Extensive results demonstrate that FSCOI has a better performance than OSA.

Performance Evaluation of a Power-Efficient and Robust 60 GHz Wireless Server-to-Server Datacenter Network

Datacenters have become the digital backbone of the modern world. To support the growing demands on bandwidth, datacenter networks (DCNs) consume an increasing amount of power. Additionally, the complex cabling in traditional datacenters poses design and maintenance challenges and increases the energy cost of the cooling infrastructure by obstructing the flow of chilled air. In this paper, we present a wireless server-to-server datacenter network architecture using millimeter-wave links to eliminate the need for power-hungry switching fabric of traditional fat-tree based datacenter networks. The server-to-server wireless datacenter network (S2S-WiDCN) architecture requires line-of-sight (LoS) between servers to establish direct communication links. However, in the presence of an obstruction such as an IT technician, the LoS may be blocked. To address this issue, we propose a novel obstruction-aware adaptive routing algorithm for S2S-WiDCN. We evaluate the performance of S2S-WiDCN in terms of traffic flow completion duration and throughput, with different kinds of datacenter traffic as well as in presence of obstruction to LoS links. We also compare the performance and power efficiency of S2S-WiDCN with the traditional fat tree DCN as well as a top-of-rack (ToR)-to-ToR wireless DCN.

Tomogravity space based traffic matrix estimation in data center networks

Traffic matrix (TM) is an important input requirement to better system management in data center networks (DCNs). Directly estimating TM is cost and difficult since the flow behaviors in DCNs are irregular and the TM across the Top of Rack (ToR) switches is huge. Although indirect TM estimation tomography based methods can be applied in DCNs after decomposing tree-like structure, these approaches require a good prior TM obtained by gravity model to improve estimation accuracy. In addition, data collection from the Simple Network Management Protocol (SNMP) employed in DCNs can result in unavoidable data missing and data errors. Therefore, it is necessary to estimate a good prior TM and study the effect of link data missing or data errors on estimation accuracy. In this paper, we utilize the tomogravity space to achieve TM estimation in decomposed tree-like DCNs without requiring a good prior TM because the gravity model can be replaced by gravity space. We propose two iterative algorithms to estimate TM between tomogravity space and gravity space, and use similar-Mahalanobis distance as a metric to control estimation errors. One iterative algorithm utilizes a prior TM calculated based on coarse-grained traffic characteristics, whereas the other considers moderate link data missing and no prior TM based on traffic characteristics. To further separately discuss the effect of link data errors, we obtain desirable link measurement from packet trace and routing matrix. Numerical results demonstrate that our iterative algorithms outperform the existing algorithms in terms of controlling data errors based on decomposed structure and produce robust results when adding different noise level on the link data.

Optical trends in data centers architectures for smart cities

This paper aims to present a new data center topology using new components such as top of rack (ToR), switches for the aggregation of each rack, optical space switches (OSS) used for intra/inter connectivity, and software defined network (SDN) as control plane for the configuration of the associated network resources. The effect of flows streams is examined further on the performance of the data centers developed into smart cities. Moreover, a comparison between the traditional non-blocking hierarchical network and the proposed new topology all-optical in terms of completion time for multicast and unicast flows are presented.

POTORI: A Passive Optical Top-of-Rack Interconnect Architecture for Data Centers

Several optical interconnect architectures inside data centers (DCs) have been proposed to efficiently handle the rapidly growing traffic demand. However, not many works have tackled the interconnects at top-of-rack (ToR), which have a large impact on the performance of the data center networks (DCNs) and can introduce serious scalability limitations due to their high cost and power consumption. In this paper, we propose a passive optical ToR interconnect architecture (POTORI) to replace the conventional electronic packet switch (EPS) in the access tier of DCNs. In the data plane, POTORI relies on a passive optical coupler to interconnect the servers within the rack and interfaces toward the aggregation/core tiers. The POTORI control plane is based on a centralized rack controller responsible for managing the communications among the servers in the rack. We propose a cycle-based medium access control (MAC) protocol to efficiently manage the exchange of control messages and the data transmission inside the rack. We also introduce and evaluate a dynamic bandwidth allocation algorithm for POTORI, namely largest first (LF). Extensive simulation results show that, with the use of fast tunable optical transceivers, POTORI and the proposed LF strategy are able to achieve an average packet delay below 10 μs under realistic DC traffic scenarios, outperforming conventional EPSs. On the other hand, with slower tunable optical transceivers, a careful configuration of the network parameters (e.g., maximum cycle time of the MAC protocol) is necessary to obtain a good network performance in terms of the average packet delay.

Towards Petabit/s All-Optical Flat Data Center Networks Based on WDM Optical Cross-Connect Switches with Flow Control

Scaling the capacity while maintaining low latency and power consumption is a challenge for hierarchical data center networks (DCNs) based on electrical switches. In this work we present a novel all-optical flat DCN architecture OPSquare that potentially addresses the scaling issues by employing parallel intra-/inter-cluster switching networks, distributed fast WDM optical cross-connect (OXC) switches, and a novel top-of-rack (ToR) switch architecture. The fast (nanoseconds) WDM OXC switches allow flexible switching capability in both wavelength and time domains and statistical multiplexing. The OPSquare DCN performance targeting Petabit/s capacity has been thoroughly assessed. First the packet loss, latency, throughput, and scalability are numerically investigated under realistic data center traffic model. Results indicate that when scaling the DCN size up to 1024 ToR switches, a packet loss ratio below 10−6 and a server end-to-end latency lower than 2 µs can be guaranteed at load of 0.3 with limited 20 kB buffer. Then, the experimental evaluation of the DCN by employing 4 × 4 OXC prototypes shows multi-path dynamic switching with flow control operation. The case deploying 32 × 32 and 64 × 64 OXC switches connecting 1024 and 4096 ToRs are emulated and limited performance degradation has been observed. The potential of switching higher-order modulation and waveband signals further proves the suitability of OPSquare architecture for Petabit/s and low-latency DCN by using optical switches with moderate radix.

All-Optical Programmable Disaggregated Data Centre Network Realized by FPGA-Based Switch and Interface Card

This paper reports an FPGA-based switch and interface card (SIC) and its application scenario in an all-optical, programmable disaggregated data center network (DCN). Our novel SIC is designed and implemented to replace traditional optical network interface cards, plugged into the server directly, supporting optical packet switching (OPS)/optical circuit switching (OCS) or time division multiplexing (TDM)/wavelength division multiplexing (WDM) traffic on demand. Placing the SIC in each server/blade, we eliminate electronics from the top of rack (ToR) switch by pushing all the functionality on each blade while enabling direct intrarack blade-to-blade communication to deliver ultralow chip-to-chip latency. We demonstrate the disaggregated DCN architecture scenarios along with all-optical dimension-programmable N × M spectrum selective Switches (SSS) and an architecture-on-demand (AoD) optical backplane. OPS and OCS complement each other as do TDM and WDM, which can support variable traffic flows. A flat disaggregated DCN architecture is realized by connecting the optical ToR switches directly to either an optical top of cluster switch or the intracluster AoD optical backplane, while clusters are further interconnected to an intercluster AoD for scaling out.