Multiple Coflows Research Articles

Regularization-Based Coflow Scheduling in Optical Circuit Switches

To improve the application-level data efficiency, the scheduling of coflows, defined as a collection of parallel flows sharing the same objective, is prevailing in recent data centers. Meanwhile, optical circuit switches (OCS) are gradually applied to provide high data rate with low power consumption. However, so far few research outputs have covered the flow, let alone the coflow, scheduling in the context of OCS. In this work, we investigate coflow scheduling in OCS-based data centers. We first derive a novel operation called regularization processed respectively on the flow traffic demands and the flow start times, which can be efficiently implemented and reduce the circuit reconfiguration frequency dramatically. We then propose a 2-approximation algorithm, called Reco-Sin, for single coflow scheduling to minimize the coflow completion time (CCT). For multiple coflows, we derive Reco-Mul to minimize the total weighted CCT, which can transform any non-preemptive multi-coflow scheduling in packet switches to a scheduling scheme in OCS. Reco-Mul can achieve a constant approximation under the assumption that no tiny flows will be transmitted in OCS. To get rid of this assumption, we present another multiple coflow scheduling scheme, named Reco-Mul+, which has an approximation ratio of O(K). Here, K is the total number of coflows. Extensive simulations based on Facebook data traces show that our approaches outperform state-of-the-art schemes significantly, i.e., one single coflow can be finished up to 1.97× faster with Reco-Sin, and multiple coflows can be completed up to more than 2× faster with Reco-Mul and Reco-Mul+.

Coflow Scheduling in Data Centers: Routing and Bandwidth Allocation

In distributed computing frameworks like MapReduce, Spark, and Dyrad, a coflow is a set of flows transferring data between two stages of a job. The job cannot start its next stage unless all flows in the coflow finish. To improve the execution performance of such a job, it is crucial to reduce the completion time of a coflow, as it can contribute more than 50 percent of the job completion time. While several coflow schedulers have been proposed, we observe that routing, as a factor greatly impacting the Coflow Completion Time (CCT), has not been well considered. In this article, we focus on the coflow scheduling problem and jointly consider routing and bandwidth allocation. We begin by providing an analytical solution to the problem of optimal bandwidth allocation with pre-determined routes. In the following, we formulate the problem of scheduling a single coflow as a Non-linear Mixed Integer Programming problem and present its relaxed convex optimization problem. We further propose two algorithms, CoRBA and its simplified version: CoRBA-fast that solve the single coflow scheduling problem with a joint consideration of routing and bandwidth allocation. Lastly, to address multiple coflows in online scheduling, we propose an online scheduler named OnCoRBA. By comparing with the start-of-the-art algorithms and schedulers via simulations, we demonstrate that CoRBA and CoRBA-fast reduce the CCT by 30-400 percent and the OnCoRBA scheduler reduces the average online CCT by 20-230 percent. In addition, CoRBA-fast can be hundreds times faster than CoRBA with around 8 percent performance degradation compared to CoRBA, which makes the use of CoRBA-fast very appropriate in practice.

Joint Coflow Optimization for Data Center Networks

Data parallel applications in data centers generate, process, and store huge volumes of data. Coflow Completion time (CCT) is one of the major performance metrics to capture application-level semantics. This paper is the first one to study the joint consideration of task placement, coflow bandwidth scheduling, and path choice to minimize the average CCT in intra-data center. This paper proposes a joint online scheduling framework, which first develops a 2-approximation algorithm to reduce the CCT of a single coflow, and then follows the Shortest Remaining Time First (SRTF) principle to schedule multiple coflows. Extensive simulations based on practical trace demonstrate that the proposed framework has better performance than the state-of-the-art works.

Joint coflow routing and scheduling in leaf-spine data centers

Communication in data centers often involves many parallel flows that all share the same performance goal (e.g. to minimize the average completion time). A useful abstraction, coflow, is proposed to express the communication requirements of prevalent data parallel paradigms such as MapReduce and Spark. The multiple coflow routing and scheduling problem makes it challenging to derive a good theoretical performance ratio, as coexisting coflows may compete for the same network resources such as link bandwidths. In this paper, we focus on the coflow problem in one popular data center infrastructure: the Leaf-Spine topology. We first formulate the problem and study the path selection issue on this two-tier structure. In order to minimize the average coflow completion time (CCT), we propose the Multi-hop Coflow Routing and Scheduling strategy (MCRS) and prove that our method has a reasonably good competitive ratio. Extensive experiments and large-scale simulations show that MCRS outperforms the state-of-the-art heuristic schemes under the Leaf-Spine topology.

Joint Online Coflow Optimization Across Geo-Distributed Datacenters

Growing data volumes have been generated, stored and processed across geographically distributed datacenters. Processing such geo-distributed datasets should consider task or job placement, heterogeneous available link bandwidth, and routing decisions. Thus optimizing the transmissions of inter-datacenter flows, especially coflows that capture application-level semantics, is important. However, prior solutions on coflow scheduling either merely ignore routing or fix the endpoint placement. This article focuses on the problem of jointly considering endpoint placement, coflow scheduling, and coflow routing to minimize the average coflow completion time (CCT) across geo-distributed datacenters. This paper first presents the system model and problem formulation, then derives a randomized approximation algorithm for a single coflow. After that it also proposes an online algorithm to handle the complex cases of multiple coflows. Through theoretical analysis, a proof that the proposed algorithms have a non-trivial competitive ratio is given. Results from extensive simulations demonstrate that the proposed algorithms can significantly reduce the CCT of coflows and at the same time, have similar algorithm run time, compared with the state-of-the-art solutions.

Joint Online Coflow Routing and Scheduling in Data Center Networks

A coflow is a collection of related parallel flows that occur typically between two stages of a multi-stage computing task in a network, such as shuffle flows in MapReduce. The coflow abstraction allows applications to convey their semantics to the network so that application-level requirements can be better satisfied. In this paper, we study the routing and scheduling of multiple coflows to minimize the total weighted coflow completion time (CCT). We first propose a rounding-based randomized approximation algorithm, called OneCoflow , for single coflow routing and scheduling. The multiple coflow problem is more challenging as coexisting coflows will compete for the same network resources, such as link bandwidth. To minimize the total weighted CCT, we derive an online multiple coflow routing and scheduling algorithm, called OMCoflow . We then derive a competitive ratio bound of our problem and prove that the competitive ratio of OMCoflow is nearly tight. To the best of our knowledge, this is the first online algorithm with theoretical performance guarantees which considers routing and scheduling simultaneously for multi-coflows. Compared with existing methods, OMCoflow runs more efficiently and avoids frequently rerouting the flows. Extensive simulations on a Facebook data trace show that OMCoflow outperforms the state-of-the-art heuristic schemes significantly (e.g., reducing the total weighted CCT by up to 41.8% and the execution time by up to 99.2% against RAPIER).

Joint optimization of tasks placement and routing to minimize Coflow Completion Time

Coflow is defined as the parrellel flows between two successive computation stages of data-parallel jobs. Reducing Coflow Completion Time (CCT) is important to improve the performance of data-parallel applications in geo-distributed datacenter networks. CCT is influenced by two factors: tasks placement and coflow routing. Previous works that optimize only one factor are insufficient in reducing CCT.In this paper, we explore the joint optimization of tasks placement and coflow routing to reduce CCT. We formulate the joint optimization of a single coflow as a Mixed Integer Non-Linear Programming problem, and propose an approximate algorithm PRO with an approximation ratio of (1 + ε). Moreover, we propose algorithms to optimize multiple coflows scheduling offline and online. Through extensive experiments, we demonstrate that our proposed algorithms have superior performance in reducing CCT or average CCT compared with other algorithms.

Engineering a material for biomedical applications with electric field assisted processing

In this work, using multiple co-flows we demonstrate in-situ encapsulation of nano-particles, liquids and/or gases in different structural morphologies, which can also be deposited in a designated pattern by a direct write method and surface modification can be controlled to release encapsulated material. The range of possibilities offered by exposing a material solution to an applied electric field can result in a plethora of structures which can accommodate a whole host of biomedical applications from microfluidic devices (microchannels, loaded with various materials), printed 3D structures and patterns, lab-on-a-chip devices to encapsulated materials (capsules, tubes, fibres, dense multi-layered fibrous networks) for drug delivery and tissue engineering. The structures obtained in this way can vary in size from micrometer to the nanometer range and the processing is viable for all states of matter. The work shown demonstrates some novel structures and methodologies for processing a biomaterial.