Optical Circuit Switching Research Articles

Nonblocking conditions for Clos fabrics with non-uniform switch radixes

Datacenter networks (DCNs) evolve over years and so comprise switches from different generations. Thus, each stage/layer of the Clos fabric may consist of switches with varying radixes (i.e., different port counts), leading to non-uniform stages. While optical circuit switches are increasingly deployed in DCNs to enhance transmission capacity and energy efficiency, the nonblocking condition, crucial for determining the performance of circuit-switched networks, has been established only for Clos fabrics with uniform stages. This study extends the nonblocking condition to Clos fabrics with non-uniform stages. To facilitate practicality, we formulate the condition using integer linear programming (ILP). Using our novel, to our knowledge, condition, we quantitatively demonstrate how much the nonblocking property is compromised under two practical scenarios, random link failures and network expansion, which would break network uniformity. In particular, we reveal that network expansion, common in DCN evolution, could significantly undermine the nonblocking property. Additionally, we assess the computational efficiency of our ILP formulation, which can successfully evaluate the nonblocking property of a large Clos fabric accommodating 32K terminals/uplinks in just 19 min.

Optical circuit switched three-stage twisted-folded Clos-network design model guaranteeing admissible blocking probability

Some data center networks have already started to use optical circuit switching (OCS) with potential performance benefits, including high capacity, low latency, and energy efficiency. This paper addresses a switching network design to maximize the network radix, i.e., the number of terminals connected to the network under the condition that a specified number of identical switches with the size N×N and the maximum admissible blocking probability are given. Previous work presented a two-stage twisted and folded Clos network (TF-Clos) with a blocking probability guarantee for OCS, which has a larger network radix than TF-Clos with a strict-sense non-blocking condition. Expanding the number of stages allows for enhancing the network radix. This paper proposes a model designing an OCS three-stage TF-Clos structure with a blocking probability guarantee to increase the network radix compared to the two-stage TF-Clos. We formulate the problem of obtaining the network configuration that maximizes the network radix as an optimization problem. We conduct an algorithm based on an exhaustive search to obtain a feasible solution satisfying the constraints of the optimization problem. This algorithm identifies the structure with the largest network radix in non-increasing order to avoid unnecessary searches. Numerical results show that the proposed model achieves a larger network radix than the two-stage model.

COCSN: A Multi-Tiered Cascaded Optical Circuit Switching Network for Data Center

COCSN: A Multi-Tiered Cascaded Optical Circuit Switching Network for Data Center

Dynamic capacity sharing with multi-wavelength integrated transmitters in hybrid datacenter networks

In datacenters, bursty and unevenly distributed traffic may lead to serious network performance degradation. Various methods, including reconfigurable optical circuit switching (OCS), traffic control techniques, valiant load balancing (VLB), and so on, have been proposed to solve this problem. Based on these solutions, our method makes a trade-off between cost and performance. In this paper, we propose to use multi-wavelength tunable transmitters in our previously proposed modular arrayed waveguide grating (AWG)-based interconnection network. We discuss how the multiple wavelengths can be shared in the network and then propose a computational model to study its blocking probability. Closed-form equations for low network load cases are also derived to provide the analytical expression for the blocking probability. We verify the accuracy of our computational model through simulations. Comparing the blocking probability of networks with and without multi-wavelength integrated transmitters, we show that network performance can be considerably improved after replacement. When traffic burstiness is 1.25 and traffic skewness is 0.08, the blocking probability is reduced from 0.14 to 3.60×10−3 after replacing in each sending module one fixed laser with multi-wavelength tunable transmitters with four wavelengths. Furthermore, we also discuss how different factors influence the blocking probability and the maximum load with the given network performance requirement.

Efficient fiber-inspection and certification method for optical-circuit-switched datacenter networks

Datacenter networks (DCNs) consisting of optical circuit switches (OCSs) have been considered as a promising solution to dramatically improve their transmission capacity, energy efficiency, and communication latency. To scale optical-circuit-switched DCNs (OCS DCNs), hierarchical OCSs with tens of thousands of optical fibers need to be installed, and they should be inspected before starting datacenter operations. Since traditional DCNs consist of electrical-packet switches (EPSs), the condition and cabling of fibers can be inspected easily by probing neighboring EPSs. However, OCS networks cannot be inspected in the same manner because OCSs cannot transmit and receive probe signals. Thus, we have had to attach and detach a light source and power meter (LSPM) to every switch for probing all the fibers, which takes weeks. This paper proposes an efficient method for inspecting and certifying fibers in an entire DCN without repeating LSPM reattachment. Our method is based on (1) theories on quickly estimating the fiber condition on the basis of the intensity of received probe signals, (2) the maximum allowable loss of each fiber derived from the transceiver budget used in operations, and (3) an algorithm that reduces the number of probes needed. The results from an extensive numerical evaluation indicate that our method inspected a DCN with 18,432 fibers in at most a day, whereas a baseline method involving repeated LSPM reattachment would take more than a week. We also confirmed that our method never produced false negatives and false positives under practical network conditions.

Orchid: enhancing HPC interconnection networks through infrequent topology reconfiguration

Interconnection networks are key components of high-performance computing (HPC) systems. As HPC evolves towards the exascale era, providing sufficient bisection bandwidth between computing node pairs through oversubscription in traditional networks becomes prohibitively expensive and impractical. Over the past decade, several architectures leveraging optical circuit switches (OCSs) for dynamic link bandwidth allocation have gained traction. These architectures require frequent network topology reconfiguration to adapt to changing traffic demands. However, practical implementation remains hampered by the long reconfiguration delays inherent in OCS technology. We propose Orchid, an architecture that leverages OCSs to achieve infrequent topology reconfigurations, effectively addressing the problem of long reconfiguration delays. A key innovation of Orchid is its ability to extract stable traffic matrices from historical data. This functionality guides the reconfiguration of the topology without the need for adjustments with each traffic matrix, thereby enabling the sharing of OCS overhead over an extended timeframe. Furthermore, Orchid addresses potential congestion arising from unexpected traffic through the joint design of OCS configuration and routing, ensuring an even distribution of traffic across global links. Extensive experiments using real HPC application traces and synthetic traffic demonstrate that Orchid achieves significant performance improvements compared to existing HPC interconnection networks. Specifically, Orchid reduces packet delay by at least 3× and enhances throughput by up to 60%.

7424 × 7424 Optical Circuit Switch With 1.4 μs Switching Time Enabled by Silicon-Photonic 64-Port Space Switches and Burst-Mode Coherent Receivers

7424 × 7424 Optical Circuit Switch With 1.4 μs Switching Time Enabled by Silicon-Photonic 64-Port Space Switches and Burst-Mode Coherent Receivers

Next Generation Exabit/s Data Center Network Supporting All Optical Circuit and Packet Switching

Next Generation Exabit/s Data Center Network Supporting All Optical Circuit and Packet Switching

Optical switching will innovate intra data center networks [Invited Tutorial

Reflecting the recent slow-down in Moore’s law and the proliferation of artificial intelligence/machine learning workloads, the performance and energy consumption of networks are becoming barriers in high-performance computing (HPC) and data centers. Optical switches are expected to break these barriers, and indeed their introduction has recently commenced in data centers. This paper discusses how optical switching technologies can innovate future intra data center networks. Hyperscale data centers are much bigger in scale, and network requirements are slightly different from those of HPC. This paper focus on data center networks, since the impact of optical technologies will be more significant in data centers than in HPC. In addition to the scale issue, important metrics to be considered for network design are traffic characteristics and latency, both of which are highlighted in this paper. For hybrid (electrical packet and optical circuit) switching networks, the target latency for the optical circuit switch network (connection setup/teardown time) is shown to be around 10 µs, and the needed technologies are clarified and verified by experiments. The optical switch can simplify the present multi-tier switching network above tier-1 switches into a single tier configuration, which is possible with the development of efficient large port count optical switches. Among the different switching architectures, combining the different dimensions of space and wavelength is shown to be one of the best solutions. Fast switching needs fast device response time. Si photonics devices using Mach–Zehnder interferometers or ring-resonator-based switches and tunable filters are the most promising candidates; they offer cost-effective mass-production and fast operation and so are excellent candidates for the optical switches envisaged. Another critical technology to maximize the benefits of optical switches is a simple and low-latency control mechanism. Different approaches have been suggested as summarized in this work. Among them, harnessing optical switch parallelism is a unique technique that matches recent advances in electrical switch chips. A fast control network is realized by using a fully decentralized and asynchronous control mechanism. A hyperscale data center offers a wide variety of services, and no one system fits all needs. Optimization of parameters is an important task for maximizing the impact of optical switching in different kinds of data centers.

RAMP: A flat nanosecond optical network and MPI operations for distributed deep learning systems

RAMP: A flat nanosecond optical network and MPI operations for distributed deep learning systems

Load-optimization in Reconfigurable Data-center Networks: Algorithms and Complexity of Flow Routing

Emerging reconfigurable data centers introduce unprecedented flexibility in how the physical layer can be programmed to adapt to current traffic demands. These reconfigurable topologies are commonly hybrid, consisting of static and reconfigurable links, enabled by e.g., an Optical Circuit Switch (OCS) connected to top-of-rack switches in Clos networks. Even though prior work has showcased the practical benefits of hybrid networks, several crucial performance aspects are not well understood. For example, many systems enforce artificial segregation of the hybrid network parts, leaving money on the table. In this article, we study the algorithmic problem of how to jointly optimize topology and routing in reconfigurable data centers, in order to optimize a most fundamental metric, maximum link load. The complexity of reconfiguration mechanisms in this space is unexplored at large, especially for the following cross-layer network-design problem: given a hybrid network and a traffic matrix, jointly design the physical layer and the flow routing in order to minimize the maximum link load. We chart the corresponding algorithmic landscape in our work, investigating both un-/splittable flows and (non-)segregated routing policies. A topological complexity classification of the problem reveals NP-hardness in general for network topologies that are trees of depth at least two, in contrast to the tractability on trees of depth one. We moreover prove that the problem is not submodular for all these routing policies, even in multi-layer trees. However, networks that can be abstracted by a single packet switch (e.g., nonblocking Fat-Tree topologies) can be optimized efficiently, and we present optimal polynomial-time algorithms accordingly. We complement our theoretical results with trace-driven simulation studies, where our algorithms can significantly improve the network load in comparison to the state-of-the-art.

Mars: Near-Optimal Throughput with Shallow Buffers in Reconfigurable Datacenter Networks

The performance of large-scale computing systems often critically depends on high-performance communication networks. Dynamically reconfigurable topologies, e.g., based on optical circuit switches, are emerging as an innovative new technology to deal with the explosive growth of datacenter traffic. Specifically, periodic reconfigurable datacenter networks (RDCNs) such as RotorNet (SIGCOMM 2017), Opera (NSDI 2020) and Sirius (SIGCOMM 2020) have been shown to provide high throughput, by emulating a complete graph through fast periodic circuit switch scheduling. However, to achieve such a high throughput, existing reconfigurable network designs pay a high price: in terms of potentially high delays, but also, as we show as a first contribution in this paper, in terms of the high buffer requirements. In particular, we show that under buffer constraints, emulating the high-throughput complete graph is infeasible at scale, and we uncover a spectrum of unvisited and attractive alternative RDCNs, which emulate regular graphs, but with lower node degree than the complete graph. We present Mars, a periodic reconfigurable topology which emulates a d-regular graph with near-optimal throughput. In particular, we systematically analyze how the degree~d can be optimized for throughput given the available buffer and delay tolerance of the datacenter. We further show empirically that Mars achieves higher throughput compared to existing systems when buffer sizes are bounded.

Enabling Quasi-Static Reconfigurable Networks With Robust Topology Engineering

Many optical circuit switched data center networks (DCN) have been proposed in the last decade to attain higher capacity and topology reconfigurability, though commercial adoption of these architectures have been minimal. One major challenge these architectures face is the difficulty of handling uncertain traffic demands using commercial optical circuit switches (OCS) with high switching latency. Prior works have generally focused on developing fast-switching OCS prototypes to quickly react to traffic variations through frequent reconfigurations. This approach, however, adds tremendous complexity overhead to the control plane, and raises the barrier for commercial adoption of optical circuit switched data center networks. We propose, a robust topology and routing optimization framework for reconfigurable optical circuit switched data centers. co-optimizes topology and routing based on a convex set of traffic matrices, and offers strict throughput guarantees for any future traffic matrices bounded by the convex set. For the bursty traffic demands that are unbounded by the convex set, we employ a desensitization technique to reduce performance hit. This enables to generate topology and routing solutions capable of handling unexpected traffic changes without relying on frequent topology reconfigurations. Our extensive evaluations based on Facebook’s production DCN traces show that, even with daily reconfigurations which could be realized by current commercial MEMS-based OCSs from Calient Technologies, achieves about 20% lower max link utilization, and about 32% lower average hop count compared to cost-equivalent static topologies. Our work shows that adoption of reconfigurable topologies in commercial DCNs is feasible even without fast OCSs.

Multicore fiber interconnects for multi-terabit spine-leaf datacenter network topologies

This work demonstrates the use of spatial division multiplexing (SDM) to implement spine-leaf datacenter network topologies. We consider two possible scenarios, with the first consisting of the direct use of high-capacity multicore fiber interconnects. This was implemented with an 8-core multicore fiber to support 64×200Gb/s SDM wavelength division multiplexing (WDM) lanes and combined throughput of 12.8 Tb/s. We also demonstrate a scenario in which each multicore fiber interconnect is used to establish multiple connections between pairs of spin-leaf switches. An optical circuit switch is used between the spine and leaf layers to establish core-selective connections, which significantly reduces the number of cables required to implement the network. We demonstrate this approach with a commercial optical circuit switch and 8-core and 4-core multicore fiber interconnects.

Mars: Near-Optimal Throughput with Shallow Buffers in Reconfigurable Datacenter Networks

The performance of large-scale computing systems often critically depends on high-performance communication networks. Dynamically reconfigurable topologies, e.g., based on optical circuit switches, are emerging as an innovative new technology to deal with the explosive growth of datacenter traffic. Specifically, periodic reconfigurable datacenter networks (RDCNs) such as RotorNet (SIGCOMM 2017), Opera (NSDI 2020) and Sirius (SIGCOMM 2020) have been shown to provide high throughput, by emulating a complete graph through fast periodic circuit switch scheduling. However, to achieve such a high throughput, existing reconfigurable network designs pay a high price: in terms of potentially high delays, but also, as we show as a first contribution in this paper, in terms of the high buffer requirements. In particular, we show that under buffer constraints, emulating the high-throughput complete graph is infeasible at scale, and we uncover a spectrum of unvisited and attractive alternative RDCNs, which emulate regular graphs, but with lower node degree than the complete graph. We present Mars, a periodic reconfigurable topology which emulates ad-regular graph with near-optimal throughput. In particular, we systematically analyze how the degree d can be optimized for throughput given the available buffer and delay tolerance of the datacenter. We further show empirically that Mars achieves higher throughput compared to existing systems when buffer sizes are bounded.

Duo: A High-Throughput Reconfigurable Datacenter Network Using Local Routing and Control

The performance of many cloud-based applications critically depends on the capacity of the underlying datacenter network. A particularly innovative approach to improve the throughput in datacenters is enabled by emerging optical technologies, which allow to dynamically adjust the physical network topology, both in an oblivious or demand-aware manner. However, such topology engineering, i.e., the operation and control of dynamic datacenter networks, is considered complex and currently comes with restrictions and overheads. We present Duo, a novel demand-aware reconfigurable rack-to-rack datacenter network design realized with a simple and efficient control plane. Duo is based on the well-known de Bruijn topology (implemented using a small number of optical circuit switches) and the key observation that this topology can be enhanced using dynamic (''opportunistic'') links between its nodes. In contrast to previous systems, Duo has several desired features: i) It makes effective use of the network capacity by supporting integrated and multi-hop routing (paths that combine both static and dynamic links). ii) It uses a work-conserving queue scheduling which enables out-of-the-box TCP support. iii) Duo employs greedy routing that is implemented using standard IP longest prefix match with small forwarding tables. And iv) during topological reconfigurations, routing tables require only local updates, making this approach ideal for dynamic networks. We evaluate Duo in end-to-end packet-level simulations, comparing it to the state-of-the-art static and dynamic networks designs. We show that Duo provides higher throughput, shorter paths, lower flow completion times for high priority flows, and minimal packet reordering, all using existing network and transport layer protocols. We also report on a proof-of-concept implementation of Duo's control and data plane.

Shufflecast: An Optical, Data-Rate Agnostic, and Low-Power Multicast Architecture for Next-Generation Compute Clusters

An optical circuit-switched network core has the potential to overcome the inherent challenges of a conventional electrical packet-switched core of today’s compute clusters. As optical circuit switches (OCS) directly handle the photon beams without any optical-electrical-optical (O/E/O) conversion and packet processing, OCS-based network cores have the following desirable properties: a) agnostic to data-rate, b) negligible/zero power consumption, c) no need of transceivers, d) negligible forwarding latency, and e) no need for frequent upgrade. Unfortunately, OCS can only provide point-to-point (unicast) circuits. They do not have built-in support for one-to-many (multicast) communication, yet multicast is fundamental to a plethora of data-intensive applications running on compute clusters nowadays. In this paper, we propose Shufflecast, a novel optical network architecture for next-generation compute clusters that can support high-performance multicast satisfying all the properties of an OCS-based network core. Shufflecast leverages small fanout, inexpensive, passive optical splitters to connect the Top-of-rack (ToR) switch ports, ensuring data-rate agnostic, low-power, physical-layer multicast. We thoroughly analyze Shufflecast’s highly scalable data plane, light-weight control plane, and graceful failure handling. Further, we implement a complete prototype of Shufflecast in our testbed and extensively evaluate the network. Shufflecast is more power-efficient than the state-of-the-art multicast mechanisms. Also, Shufflecast is more cost-efficient than a conventional packet-switched network. By adding Shufflecast alongside an OCS-based unicast network, an all-optical network core with the aforementioned desirable properties supporting both unicast and multicast can be realized.

Topology configuration scheme for accelerating coflows in a hyper-FleX-LION

To address the challenges that data center networks (DCNs) are currently facing, optical circuit switching in DCNs has been introduced and a number of architectures to improve the performance of DCNs in terms of port capacity, energy efficiency, and data transfer latency have been proposed. This led to a few all-optical interconnects (AOIs), among which Hyper-FleX-LION possesses the reconfigurability that can efficiently support various traffic patterns. In this work, we study the topology management of AOIs in Hyper-FleX-LION to efficiently schedule coflows in them so the average coflow completion time (CCT) can be minimized. We first propose time-efficient algorithms to address the provisioning of a single coflow and then extend the algorithms for multi-coflow scenarios. The simulation results demonstrate that AOIs in Hyper-FleX-LION can accelerate coflows better than the traditional AOIs based on optical cross-connects and achieve an acceleration of up to 5.6 × under the assumed situations. Our simulations also verify that, for coflow scheduling in Hyper-FleX-LION, our proposal can reduce the average CCT and significantly outperform benchmarks.

Co-Scheduler: A Coflow-Aware Data-Parallel Job Scheduler in Hybrid Electrical/Optical Datacenter Networks

To support higher demand for datacenter networks, networking researchers have proposed hybrid electrical/optical datacenter networks (Hybrid-DCN) that leverages optical circuit switching (OCS) along with traditional electrical packet switching (EPS). However, due to the high <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">reconfiguration</i> delay of OCS, OCS is used only for bulk data transfers between racks to amortize the reconfiguration delay. Existing job schedulers for data-parallel frameworks are not designed for Hybrid-DCN, since they neither place tasks to aggregate data traffic to take advantage of OCS, nor schedule tasks to minimize the Coflow completion time (CCT). In this paper, we describe the mismatch between existing job schedulers and the advanced Hybrid-DCN, introduce the requirements for the new scheduler, and present the implementation of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Co-scheduler</i> , a job scheduler for data-parallel frameworks that aims to improve job performance by placing the tasks of jobs to aggregate enough data traffic to better leverage OCS to minimize the CCT in Hybrid-DCN. Specifically, for every job, Co-scheduler computes <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">guidelines</i> on how many racks to place the job’s input data and the job’s tasks. The guidelines are dynamically generated based on the real-time job characteristics or predictable job characteristics from prior runs, with the aim of leveraging OCS whenever possible and efficient and minimizing CCT of jobs. Co-scheduler then schedules the tasks of jobs based on the guidelines. We evaluate the effectiveness of Co-scheduler using trace-driven simulation. The evaluation demonstrates that Co-scheduler can improve makespan, average job completion time, and average CCT of a workload by up to 56%, 61%, and 79%, respectively, compared to the state-of-the-art schedulers.

OSDL: Dedicated optical slice provisioning in support of distributed deep learning

OSDL: Dedicated optical slice provisioning in support of distributed deep learning