Data Center Network Architecture Research Articles

On the Network Power Effectiveness of Data Center Architectures

Cloud computing not only requires high-capacity data center networks to accelerate bandwidth-hungry computations, but also causes considerable power expenses to cloud providers. In recent years many advanced data center network architectures have been proposed to increase the network throughput, such as Fat-Tree [1] and BCube [2] , but little attention has been paid to the power efficiency of these network architectures. This paper makes the first comprehensive comparison study for typical data center networks with regard to their Network Power Effectiveness (NPE ), which indicates the end-to-end bps per watt in data transmission and reflects the tradeoff between power consumption and network throughput. We take switches, server NICs and server CPU cores into account when evaluating the network power consumption. We measure NPE under both regular routing and power-aware routing, and investigate the impacts of topology size, traffic load, throughput threshold in power-aware routing, network power parameter as well as traffic pattern. The results show that in most cases Flattened Butterfly possesses the highest NPE among the architectures under study, and server-centric architectures usually have higher NPEs than Fat-Tree and VL2 architectures. In addition, the sleep-on-idle technique and power-aware routing can significantly improve the NPEs for all the data center architectures, especially when the traffic load is low. We believe that the results are useful for cloud providers, when they design/upgrade data center networks or employ network power management.

FleCube: A flexibly-connected architecture of data center networks on multi-port servers

Underlying network provides infrastructures for cloud computing in data centers. The server-centric architectures integrate network and compute, which place routing intelligence on servers. However, the existing multi-port server based architectures suffer from determined scale and large path length. In this paper, we propose FleCube, a flexibly-connected architecture on multi-port servers without using any switches. FleCube is recursively constructed on division of multiple ports in a server by means of complete graph. FleCube benefits data center networks by flexible scale and low diameter, as well as large bisection width and small bottleneck degree. Furthermore, we develop multi-path routing (MPR) to take advantage of parallel paths between any two servers. MPR adopts random forwarding to distribute traffic load and relieve network congestion. Analysis and comparisons with existing architectures show the advantages of FleCube. Evaluations under different degrees of network traffic demonstrate the merits of FleCube and the proposed routings.

Elastic optical ring with flexible spectrum ROADMs: An optical switching architecture for future data center networks

Data center networks (DCN) are facing significant increase in both scale and bandwidth requirement, and various intra-DCN optical circuit switching architectures have been proposed to handle the inter-rack traffic. As the size of data centers are becoming larger, ring based architectures have received considerable interests, because a ring can provide connectivity to a large number of racks that are distributed far apart in a data center. However, due to the specific data center traffic pattern, fixed spectrum allocation scheme in existing ring based DCNs will lead to low spectrum utilization and hence low system throughput. In this paper, we propose to introduce flexible spectrum allocation into ring based intra-DCN optical switching, so that spectrum utilization can be improved. We design an elastic ring intra-DCN optical circuit switching architecture with flexible spectrum reconfigurable optical add-drop multiplexers (FS-ROADMs). Adopting flexible spectrum allocation in elastic ring, a spectrum allocation problem which is essentially a very complex yet interesting optimization problem arises. For a given set of inter-rack traffic requests, we develop an integer linear program formulation as well as a heuristic algorithm to solve the flexible spectrum allocation problem. We study how the inter-rack traffic pattern may affect the system performances in both fixed and elastic ring cases. For elastic ring, we also analyze and evaluate the computational complexity in solving the spectrum allocation problem. Our results demonstrate that, to handle uneven inter-rack traffic pattern in DCNs, elastic ring can remarkably improve system throughput, though its computational complexity of spectrum allocation problem is slightly higher than that of the fixed ring.

EXR: Greening Data Center Network with Software Defined Exclusive Routing

The explosive expansion of data center sizes aggravates the power consumption and carbon footprint, which has restricted the sustainable growth of cloud services and seriously troubled data center operators. In recent years, plenty of advanced data center network architectures have been proposed. They usually employ richly-connected topologies and multi-path routing to provide high network capacity. Unfortunately, they also undergo inefficient network energy usage during the traffic valley time. To address the problem, many energy-aware flow scheduling algorithms are proposed recently, primarily considering how to aggregate traffic by flexibly choosing the routing paths, with flows fairly sharing the link bandwidths. In this paper, we leverage software defined network (SDN) technique and explore a new solution to energy-aware flow scheduling, i.e., scheduling flows in the time dimension and using exclusive routing (EXR) for each flow, i.e., a flow always exclusively utilizes the links of its routing path. The key insight is that exclusive occupation of link resources usually results in higher link utilization in high-radix data center networks, since each flow does not need to compete for the link bandwidths with others. When scheduling the flows, EXR leaves flexibility to operators to define the priorities of flows, e.g., based on flow size, flow deadline, etc. Extensive simulations and testbed experiments both show that EXR can effectively save network energy compared with the regular fair-sharing routing (FSR), and significantly reduce the average flow completion time if assigning higher scheduling priorities to smaller flows.

Blocking probability in optical interconnects in data center networks

Cloud computing and Web-based applications are creating a need for powerful data centers. Data centers have a great need for high bandwidth, low latency, low blocking probability, and low bit-error rate to sustain the interaction between different applications. Current data center networks (DCNs) suffer from several problems such as high-energy consumption, high latency, fixed throughput of links, and limited reconfigurability. Electronic switches are low radix and have high latency due to a large hop count since each hop employs a store-and-forward mechanism. Optical interconnects, on the other hand, offer several advantages such as low-energy consumption, high bandwidth, reconfigurability, malleability to changing traffic, high-radix switch design, fast switching transition times, and wavelength multiplexing. These benefits provide the incentive to shift from electrical interconnects to optical interconnects in DCNs. Despite several advantages over their electrical counterparts, the performance of optical interconnects can be further improved by considering some performance parameters of optical interconnects. One such important parameter for the performance of any communication network is the blocking probability. This paper makes a comprehensive investigation of the performance of optical interconnects in different DCN architectures on the basis of blocking probability and concludes by suggesting ways to reduce the blocking.

ADON: a scalable AWG-based topology for datacenter optical network

With the development of cloud computing and other online applications, data centers holding hundreds of thousands of servers need to handle massive amounts of data and the network traffic has increased significantly. These services have driven a new design of data center networks to improve network capacity. In this paper, we propose ADON, an Arrayed Waveguide Grating Router (AWGR) based scalable data center network architecture. There are two ways to extend the ADON include adding more clusters in the same layer and adding more layers. ADON alleviates the problem of scalability. It can support more servers than Fat Tree. Moreover ADON exploits the wavelength routing characteristic of the AWGR fabric and resolves contention in the optical domain. An experimental platform based on OPNET is built to test the performance of ADON. The simulation results show that not only ADON guarantees such merits as high throughput and low latency than Fat Tree, it also improves the availability rate of the equipment and reduces the construction cost of the network appreciably compared with Petax.

Designing efficient high performance server-centric data center network architecture

Data center network (DCN) architecture is regarded as one of the most important determinants of network performance. As the most typical representatives of DCN architecture designs, the server-centric scheme stands out due to its good performance in various aspects. In this paper, we firstly present the design, implementation and evaluation of SprintNet, a novel server-centric network architecture for data centers. SprintNet achieves high performance in network capacity, fault tolerance, and network latency. SprintNet is also a scalable, yet low-diameter network architecture where the maximum shortest distance between any pair of servers can be limited by no more than four and is independent of the number of layers. The specially designed routing schemes for SprintNet strengthen its merits. However, all of these kind of server-centric architectures still suffer from some critical shortcomings owing to the server’s responsibility of forwarding packets. With regard to these issues, in this paper, we then propose a hardware based approach, named “Forwarding Unit” to provide an effective solution to these drawbacks and improve the efficiency of server-centric architectures. Both theoretical analysis and simulations are conducted to evaluate the overall performance of SprintNet and the Forwarding Unit approach with respect to cost-effectiveness, fault-tolerance, system latency, packet loss ratio, aggregate bottleneck throughput, and average path length. The evaluation results convince the feasibility and good performance of both SprintNet and Forwarding Unit.

OPMDC: Architecture Design and Implementation of a New Optical Pyramid Data Center Network

In this paper, we propose a novel optical pyramid data center network architecture (OPMDC), achieving scalable and high bandwidth, low latency, and reduced power consumption and wiring complexity. Based on an incremental and modular design, a full-scale OPMDC is built on three types of WSS-based optical switching nodes in three tiers. These optical nodes are recursively interconnected according to a pyramid structure parameterized by the number of nodes ( B ) at the base. Such a pyramid-based topology facilitates horizontal mesh connections that are tailored to achieve flexible optical packet-based transport. To this aim, we design a wavelength scheduling algorithm, called most-contentious-first, augmented with source and destination relay and aggregation (SDRA) via the horizontal mesh connections in the source and destination pods. Simulation results show that employing SDRA results in a vast improvement of throughput from 42.5% to 87.9% at full load, at a cost of no more than two additional hops of latency. Further, we present our OPMDC ( $B = 7$ ) prototype and provide performance assessment and the measurements of packet latency. Finally, experimental results show that after travelling the longest possible path with six optical nodes in OPMDC, the signal suffers no accumulated distortion and noise from the cascades of WSS and EDFA modules.

Schemes for Fast Transmission of Flows in Data Center Networks

In this paper, we survey different existing schemes for the transmission of flows in Data Center Networks (DCNs). The transport of flows in DCNs must cope with the bandwidth demands of the traffic that a large number of data center applications generates and achieve high utilization of the data center infrastructure to make the data center financially viable. Traffic in DCNs roughly comprises short flows, which are generated by the Partition/Aggregate model adopted by several applications and have sizes of a few kilobytes, and long flows, which are data for the operation and maintenance of the data center and have sizes on the order of megabytes. Short flows must be transmitted (or completed) as soon as possible or within a deadline, and long flows must be serviced with a minimum acceptable throughput. The coexistence of short and long flows may jeopardize achieving both performance objectives simultaneously. This challenge has motivated growing research on schemes for managing the transmission of flows in DCNs. We describe several recent schemes aimed at reducing the flow completion time in DCNs. We also present a summary of existing solutions for the incast traffic phenomenon. We provide a comparison and classification of the surveyed schemes, describe their advantages and disadvantages, and show the different trends for scheme design. For completeness, we describe some DCN architectures, discuss the traffic patterns of DCNs, and discuss why some existing versions of transport protocols may not be usable in DCNs. At the end, we discuss some of the identified research challenges.

Community-based complex cloud data center

The communication infrastructure is a critical component of a large-scale cloud data center. It needs to provide the best performance available while keeping overprovisioning and, lately even more important, power consumption, to the minimum. Aiming to provide a unified solution that will have high performance together with economical benefits and power consumption reduction, in this paper, we propose a new community-based scale-free model for data center network architecture. By comparing the proposed model to other similar solutions we show that the performance of the network in terms of average path length, bandwidth and resilience is similar to the state-of-the-art models. In our presented detailed analysis of the model properties, our focus is set on exploring how heterogeneity in terms of different type of network equipment influences the basic network properties. We also present solutions and network metrics that can be used in conjunction to the introduced community structure in order to additionally increase the performance.

Data center network architecture in cloud computing: review, taxonomy, and open research issues

The data center network (DCN), which is an important component of data centers, consists of a large number of hosted servers and switches connected with high speed communication links. A DCN enables the deployment of resources centralization and on-demand access of the information and services of data centers to users. In recent years, the scale of the DCN has constantly increased with the widespread use of cloud-based services and the unprecedented amount of data delivery in/between data centers, whereas the traditional DCN architecture lacks aggregate bandwidth, scalability, and cost effectiveness for coping with the increasing demands of tenants in accessing the services of cloud data centers. Therefore, the design of a novel DCN architecture with the features of scalability, low cost, robustness, and energy conservation is required. This paper reviews the recent research findings and technologies of DCN architectures to identify the issues in the existing DCN architectures for cloud computing. We develop a taxonomy for the classification of the current DCN architectures, and also qualitatively analyze the traditional and contemporary DCN architectures. Moreover, the DCN architectures are compared on the basis of the significant characteristics, such as bandwidth, fault tolerance, scalability, overhead, and deployment cost. Finally, we put forward open research issues in the deployment of scalable, low-cost, robust, and energy-efficient DCN architecture, for data centers in computational clouds.

Fastpass

An ideal datacenter network should provide several properties, including low median and tail latency, high utilization (throughput), fair allocation of network resources between users or applications, deadline-aware scheduling, and congestion (loss) avoidance. Current datacenter networks inherit the principles that went into the design of the Internet, where packet transmission and path selection decisions are distributed among the endpoints and routers. Instead, we propose that each sender should delegate control---to a centralized arbiter---of when each packet should be transmitted and what path it should follow. This paper describes Fastpass, a datacenter network architecture built using this principle. Fastpass incorporates two fast algorithms: the first determines the time at which each packet should be transmitted, while the second determines the path to use for that packet. In addition, Fastpass uses an efficient protocol between the endpoints and the arbiter and an arbiter replication strategy for fault-tolerant failover. We deployed and evaluated Fastpass in a portion of Facebook's datacenter network. Our results show that Fastpass achieves high throughput comparable to current networks at a 240x reduction is queue lengths (4.35 Mbytes reducing to 18 Kbytes), achieves much fairer and consistent flow throughputs than the baseline TCP (5200x reduction in the standard deviation of per-flow throughput with five concurrent connections), scalability from 1 to 8 cores in the arbiter implementation with the ability to schedule 2.21 Terabits/s of traffic in software on eight cores, and a 2.5x reduction in the number of TCP retransmissions in a latency-sensitive service at Facebook.

On the performance of a scalable optical switching architecture for flat intercluster data center network with centralized control

Data centers have to sustain the rapid growth of data traffic due to the increasing demand of bandwidth-hungry Internet services. The current fat tree topology causes communication bottlenecks in the server interaction process, resulting in power-hungry O-E-O conversions that limit the minimum latency and the power efficiency of these systems. As a result, recent efforts have advocated that all optical data center networks (DCNs) have the capability to adapt to traffic requirements on demand. We present the design, implementation, and evaluation of a cascaded microelectromechanical systems switches-based DCN architecture which dynamically changes its topology and link capacities, thereby achieving unprecedented flexibility to adapt to dynamic traffic patterns. We analyze it under a data center traffic model to determine its suitability for this type of environment. The proposed architecture can be scaled to 3300 input/output ports by available experimental components with low blocking probability and latency. The blocking probability and latency are about 0.03 and 72 ms at a moderate traffic load for 32 input/output ports based on our numerical results, which are much smaller than the results for 4 input/output ports which are 0.13 and 235 ms, respectively.

Software defined flexible and efficient passive optical networks for intra-datacenter communications

Facing the huge traffic challenge, optical networking shows great advantages on capacity and energy issues. However, the efficiency and flexibility are not satisfactory to the data center networks, especially for intra-datacenter communications. This article reviews the typical architectures of data center networks, and suggests that TWDM-PON system can be used in the edge layer and aggregation layer in the datacenter networks. It proposes a software defined flexible and efficient passive optical network, combining the software defined technology and network coding, for intra-datacenter communications. Network coding is applied to increase the downstream bandwidth efficiency and overcome network bottleneck, and software defined technology provides the flexibility on wavelength assignment according to traffic statistics dynamically. To increase more efficiency, a seamless DBA (S-DBA) scheme and an ONU grouping algorithm are proposed to fully utilize the upstream idle time and minimize the traffic entering into the core layer of the datacenter networks. They realize flexible scheduling and resource allocation both in time domain and wavelength domain. The experimental simulations indicate that the proposed schemes and algorithms provide low delay, good fairness, increased efficiency and network flexibility.

POXN: A New Passive Optical Cross-Connection Network for Low-Cost Power-Efficient Datacenters

Passive optical devices, characterized by low cost, zero energy consumption, and high reliability, are essential building blocks for today's telecom network infrastructure, permeating from conventional backbone transport networks towards next-generation broadband access networks. Motivated by the striking features of passive optical devices, in this paper, we seek their potential applications in emerging datacenter networks to tackle the scalability challenges arising from cost and power. Specifically, we propose passive optical cross-connection networks (POXNs) that enable cost-saving, power-efficient, and reliable communication within datacenters. To support POXNs in warehouse-scale datacenters, we address physical-layer scalability challenges by using advanced interconnection techniques. Next, we propose a distributed polling protocol to address link-layer issues that arise from the broadcast nature of the medium. The performance of our protocol is studied through analysis and simulation. In particular, we develop an analytical model to compute lower and upper bounds on the expected delay of a packet. Numerical results show that the mean packet delay is equal to the lower bound in one regime, while converges to the upper bound in the complementary regime. Results also show that our protocol can achieve high bandwidth efficiency (no less than 85% in our studied case). Additionally, we demonstrate that our protocol can embrace scheduling algorithms that support fairness and QoS. Last, we sketch the roles POXNs can play in various datacenter network architectures in terms of capital and operational cost reductions.

OSA: An Optical Switching Architecture for Data Center Networks With Unprecedented Flexibility

A detailed examination of evolving traffic characteristics, operator requirements, and network technology trends suggests a move away from nonblocking interconnects in data center networks (DCNs). As a result, recent efforts have advocated oversubscribed networks with the capability to adapt to traffic requirements on-demand. In this paper, we present the design, implementation, and evaluation of OSA, a novel Optical Switching Architecture for DCNs. Leveraging runtime reconfigurable optical devices, OSA dynamically changes its topology and link capacities, thereby achieving unprecedented flexibility to adapt to dynamic traffic patterns. Extensive analytical simulations using both real and synthetic traffic patterns demonstrate that OSA can deliver high bisection bandwidth (60%-100% of the nonblocking architecture). Implementation and evaluation of a small-scale functional prototype further demonstrate the feasibility of OSA.

Scalable Data Center Network Architecture With Distributed Placement of Optical Switches and Racks

Cloud services are fundamentally supported by data center networks (DCNs). With the fast growth of cloud services, the scale of DCNs is increasing rapidly, leading to great concern about system scalability due to multiple constraints. This paper proposes a scalable DCN architecture based on optical switching and transmission, with the distributed placement of optical switches and server racks at different nodes in a given optical network. This solves the scalability issue by relaxing power and cooling constraints and by reducing the number of (electronic) switches using high-capacity optical switches, as well as by simplifying DCN internal connections using wavelengths in the optical network. Moreover, the distributed optical switches provide service access interfaces to meet demand within areas, and thus reduce the transmission cost of the external traffic. The major concern is the additional delay and cost for remote transmissions of the DCN internal traffic. To this end, we study the component placement problem in DCNs under a given set of external demands and internal traffic patterns. By leveraging among multiple conflicting factors such as scalability and internal overhead of the DCN as well as the transmission cost of external traffic, we propose both an integer linear program and a heuristic to minimize the system cost of a DCN while satisfying all service demands in the network. This addresses both scalability and cost minimization issues from a network point of view.

On peer-assisted data dissemination in data center networks: Analysis and implementation

Data Center Networks (DCNs) are the fundamental infrastructure for cloud computing. Driven by the massive parallel computing tasks in cloud computing, one-to-many data dissemination becomes one of the most important traffic patterns in DCNs. Many architectures and protocols are proposed to meet this demand. However, these proposals either require complicated configurations on switches and servers, or cannot deliver an optimal performance. In this paper, we propose the peer-assisted data dissemination for DCNs. This approach utilizes the rich physical connections with high bandwidths and mutli-path connections, to facilitate efficient one-to-many data dissemination. We prove that an optimal P2P data dissemination schedule exists for FatTree, a speciallydesigned DCN architecture. We then present a theoretical analysis of this algorithm in the general multi-rooted tree topology, a widely-used DCN architecture. Additionally, we explore the performance of an intuitive line structure for data dissemination. Our analysis and experimental results prove that this simple structure is able to produce a comparable performance to the optimal algorithm. Since DCN applications heavily rely on virtualization to achieve optimal resource sharing, we present a general implementation method for the proposed algorithms, which aims to mitigate the impact of the potentially-high churn rate of the virtual machines.

Novel flat datacenter network architecture based on scalable and flow-controlled optical switch system

We propose and demonstrate an optical flat datacenter network based on scalable optical switch system with optical flow control. Modular structure with distributed control results in port-count independent optical switch reconfiguration time. RF tone in-band labeling technique allowing parallel processing of the label bits ensures the low latency operation regardless of the switch port-count. Hardware flow control is conducted at optical level by re-using the label wavelength without occupying extra bandwidth, space, and network resources which further improves the performance of latency within a simple structure. Dynamic switching including multicasting operation is validated for a 4 x 4 system. Error free operation of 40 Gb/s data packets has been achieved with only 1 dB penalty. The system could handle an input load up to 0.5 providing a packet loss lower that 10(-5) and an average latency less that 500 ns when a buffer size of 16 packets is employed. Investigation on scalability also indicates that the proposed system could potentially scale up to large port count with limited power penalty.

A Scalable Multicast Source Routing Architecture for Data Center Networks

This paper introduces a new scalable, efficient and stateless source routing scheme to ensure that unified unicast and multicast packets can be delivered in the Data Center Networks (DCNs). This scheme, called Code-Oriented eXplicit multicast (COXcast), is based on unicast and eXplicit multicast (Xcast). It constructs the unicast path and multicast tree by encoding the corresponding output port bitmap of each intermediate node. Using a common identifier and a node-specific key, so packets can be self-routed to multiple receivers without requiring header modification. In addition, intermediate switches/routers on the path/tree can be stateless. Compared to traditional source-based unicasting and explicit-based multicasting schemes, COXcast has lowered the processing cost, protocol overhead and delivery latency, while simplifying the deployment and management of a large number of medium-scale multicast groups, especially when applied to large-scale DCNs.