Distributed Data Centers Research Articles

Energy-aware coordinated operation strategy of geographically distributed data centers

As digital technology continues to advance rapidly, the substantial operational electricity costs associated with the extensive deployment of globally distributed internet data centers have become a significant concern. Addressing this challenging issue is crucial for the future expansion of internet data centers. In this paper, a coordinated strategy is proposed that leverages the spatial–temporal flexibility of internet data centers and their interaction with the power system. Firstly, an energy-aware internet data center loading model is introduced that incorporates multiple coupled scheduling methods to maximize the utilization of internet data centers' flexibility potential. Secondly, the proposed internet data centers loading model is integrated into a coordinated optimization model aimed at achieving higher profits through cost reduction in response to varying electricity prices across multiple regions. To facilitate the coordinated operation of geo-graphically internet data centers, an incentive profit-sharing mechanism based on Nash Bargaining theory is designed to fairly distribute coordination profits based on the contribution ratio of each data center in coordination. Through comprehensive case studies utilizing real-world datasets, the results demonstrate that the internet data center loading model, considering multiple coupled scheduling methods, can enhance internet data centers’ flexibility potential, resulting in a significant 11.73% reduction in total cost, with the coupling relationship accounting for 5.05% of this reduction. Furthermore, the proposed profit-sharing mechanism effectively fosters coordination and enthusiasm among geographically distributed internet data centers.

Spatial and thermal aware methods for efficient workload management in distributed data centers

Geographically distributed data centers provide facilities for users to fulfill the demand of storage and computations, where most of the operational cost is due to electricity consumption. In this study, we address the problem of energy consumption of cloud data centers and identify key characteristics of techniques proposed for reducing operational costs, carbon emissions, and financial penalties due to service level agreement (SLA) violations. By considering computer room air condition (CRAC) units that utilize outside air for cooling purposes as well as temperature and space-varying properties, we propose the energy cost model which takes into account temperature ranges for cooling purposes and operations of CRAC units. Then, we propose spatio-thermal-aware algorithms to manage workload using the variation of electricity price, locational outside and within the data center temperature, where the aim is to schedule the incoming workload requests with minimum SLA violations, cooling cost, and energy consumption. We analyzed the performance of our proposed algorithms and compared the experimental results with the benchmark algorithms for metrics of interest including SLA violations, cooling cost, and overall operations cost. Modeling, experiments, and verification conducted on CloudSim with realistic data center scenarios and workload traces show that the proposed algorithms result in reduced SLA violations, save between 15% to 75% of cooling cost and between 3.89% to 39% of the overall operational cost compared to the existing solutions.

DFMCloudsim: an extension of cloudsim for modeling and simulation of data fragments migration over distributed data centers

Due to the increasing volume of data for applications running on geographically distributed Cloud systems, the need for efficient data management has emerged as a crucial performance factor. Alongside basic task scheduling, the management of input data on distributed Cloud systems has become a genuine challenge, particularly with data-intensive applications. Ideally, each dataset should be stored in the same data center as its consumer tasks so as to lead to local data accesses only. However, when a given task does not need all items within one of its input datasets, sending that dataset entirely might lead to a severe time overhead. To address this concern, a data fragmentation strategy can be considered in order to partition the datasets and process them in that form. Such a strategy should be flexible enough to support any user-defined partitioning, and suitable enough to minimize the overhead of transferring the data in their fragmented form. To simulate and estimate the basic statistics of both fragmentation and migration mechanisms prior to an implementation in a real Cloud, we chose Cloudsim, with the goal of enhancing it with the corresponding extensions. Cloudsim is a popular simulator for Cloud Computing investigations. Our proposed extension is named DFMCloudsim, its goal is to provide an efficient module for implementing fragmentation and data migration strategies. We validate our extension using various simulated scenarios. The results indicate that our extension effectively achieves its main objectives and can reduce data transfer overhead by 74.75% compared to our previous work.

Allocating energy-objective aware workflow in distributed edge micro data centres

Allocating energy-objective aware workflow in distributed edge micro data centres

Geographically distributed data management to support large-scale data analysis

Nowadays, several companies prefer storing their data on multiple data centers with replication for many reasons. The data that spans various data centers ensures the fastest possible response time for customers and workforces who are geographically separated. It also provides protecting the information from the loss in case a single data center experiences a disaster. However, the amount of data is increasing at a rapid pace, which leads to challenges in storage, analysis, and various processing tasks. In this paper, we propose and design a geographically distributed data management framework to manage the massive data stored and distributed among geo-distributed data centers. The goal of the proposed framework is to enable efficient use of the distributed data blocks for various data analysis tasks. The architecture of the proposed framework is composed of a grid of geo-distributed data centers connected to a data controller (DCtrl). The DCtrl is responsible for organizing and managing the block replicas across the geo-distributed data centers. We use the BDMS system as the installed system on the distributed data centers. BDMS stores the big data file as a set of random sample data blocks, each being a random sample of the whole data file. Then, DCtrl distributes these data blocks into multiple data centers with replication. In analyzing a big data file distributed based on the proposed framework, we randomly select a sample of data blocks replicated from other data centers on any data center. We use simulation results to demonstrate the performance of the proposed framework in big data analysis across geo-distributed data centers.

Experimental research and energy saving analysis of an integrated data center cooling and waste heat recovery system

To better make use of the waste heat in distributed data centers, a new design of integrated waste heat recovery system that can meet different needs including heating for users and data center cooling is introduced. The system can switch among different operation modes including waste heat recovery mode, natural cooling mode, composite cooling mode and refrigeration mode based on the ambient temperature. The integrated waste heat recovery system is experimented in the enthalpy-difference laboratory, and the experimental results show well consistency compared with the modeling results. The operation strategies to switch among different modes are determined based on the experimental and modeling results. The paper introduced PUE (Power Usage Effectiveness) and ERE (Energy Reuse effectiveness) to comprehensively evaluate the cooling and energy reutilization effectiveness of the system. According to the evaluation result, the integrated waste heat recovery system has well performed energy effectiveness, environmental and economic effect. The ERE values of using the integrated waste heat recovery system in Chinese typical climate cities including Harbin, Beijing, Xi'an, Shanghai and Guangzhou are 0.87, 1.02, 1.04, 1.17 and 1.29 respectively. Using this system at 96 kW distributed data center instead of CRAC (computer room air conditioning) system can save 152.7 MWh electricity, 56.4 tons of standard coal, and 244.7 tons of CO2 emission in Beijing annually. The payback periods of this new system in different climate regions in China are all within 3 years.

Multi-Dimensional Resource Allocation in Distributed Data Centers Using Deep Reinforcement Learning

With the development of edge-cloud computing technologies, distributed data centers (DCs) have been extensively deployed across the global Internet. Since different users/applications have heterogeneous requirements on specific types of ICT resources in distributed DCs, how to optimize such heterogeneous resources under dynamic and even uncertain environments becomes a challenging issue. Traditional approaches are not able to provide effective solutions for multi-dimensional resource allocation that involves the balanced utilization across different resource types in distributed DC environments. This paper presents a reinforcement learning based approach for multi-dimensional resource allocation (termed as NESRL-MRM) that is able to achieve balanced utilization and availability of resources in dynamic environments. To train NESRL-MRM’s agent with sufficiently quick wall-clock time but without the loss of exploration diversity in the search space, a natural evolution strategy (NES) is employed to approximate the gradient of the reward function. To realistically evaluate the performance of NESRL-MRM, our simulation evaluations are based on real-world workload traces from Amazon EC2 and Google datacenters. Our results show that NESRL-MRM is able to achieve significant improvement over the existing approaches in balancing the utilization of multi-dimensional DC resources, which leads to substantially reduced blocking probability of future incoming workload demands.

Data Storage Optimization Model Based on Improved Simulated Annealing Algorithm

Since there is a longitudinal and horizontal penetration problem between multi-level data centers in the smart grid information transmission network. Based on the improved Simulated Annealing algorithm, this paper proposes a data storage optimization model for smart grids based on Hadoop architecture. Combining the characteristics of distributed storage in cloud computing, the smart grid data are equivalent to a task-oriented data set. The smart grid information platform is flattened, equal to a collection of multiple distributed data centers. The smart grid data over time were counted to derive the dependencies between task sets and data sets. According to the dependency between task sets and data sets, the mathematical model was established in combination with the actual data transmission of the power grid. The optimal transmission correspondence between each data set and the data center was calculated. An improved Simulated Annealing algorithm solves the longitudinal and horizontal penetration problem between multi-level data centers. When generating a new solution, the Grey Wolf algorithm provides direction for finding the optimal solution. This paper integrated the existing business data and computational storage resources in the smart grid to establish a mathematical model of the affiliation between data centers and data sets. The optimal distribution of the data set was calculated, and the optimally distributed data set was stored in a distributed physical disk. Arithmetic examples were used to analyze the efficiency and stability of several algorithms to verify the improved algorithm’s advantages, and the improved algorithms’ effectiveness was confirmed by simulation.

Coordinated operations of highly renewable power systems and distributed data centers

随着新能源并网比例的逐步提升，电网对不确定性新能源的消纳态势日趋严峻，导致弃风弃光频现。数据中心作为新兴的重要电力负荷，其运行能耗高，规模增长迅速，算力及数据资源可灵活调配，具备可挖掘的柔性调节潜力。据此，本文利用分布式数据中心集群增强电力系统柔性 调节能力，助力新能源消纳。首先，根据数据中心服务器运行工作状态和制冷设备温控要求，建立单个数据中心的精细化电力负荷模型。然后，考虑数据任务处理的时空可调节性，建立分布式数据中心互联互通的协调模型。在此基础上，提出考虑不确定性的电网 分布式数据中心集群的协同运行框架，研究集群调节能力的统一表征，建立两者顺序决策的双层优化模型，提出双层非线性区间优化方法进行求解。最后，通过算例分析验证了协同运行框架的有效性，得到了不同偏好条件下帕累托解集和分布式数据中心集群响应的变化规律。

Battery-Assisted Online Operation of Distributed Data Centers With Uncertain Workload and Electricity Prices

Battery-Assisted Online Operation of Distributed Data Centers With Uncertain Workload and Electricity Prices

Multi-Data Center Realization and Operation Efficiency Modeling Analysis of Intelligent Physical Education System

Distributed data centers shield the metadata differences of multi-source and heterogeneous remote sensing data, and provide a basis for multi-source and multi-center remote sensing data integration. Implementation in teaching and application in extracurricular sports activities, students' physical health, etc., in order to give full play to the auxiliary role of CAI in physical education, namely user management, online course management, question bank maintenance management, real-time examination management, scoring review management, System management, user health information collection, database design. When students log in to the teaching system to inquire about required materials, they can also communicate and discuss with teachers online, which promotes good communication between students and teachers. In the application process of the system, it can be found that the teaching of this system has relatively strong flexibility and is not constrained by location and time.

Comparison and Optimization of Energy Efficient Algorithm for Component Base Distributed Computing in 5G Networks

In the present scenario, the wired network or wireless networks is an application across the world. So the wire networks used in various software industries, educational institutions, and various enterprises used such as distributed data centers. The data transmission works like a flow of electricity in a linear way. So in this process during the data transmission exhibits from one stage to another the energy consumption in carbon footprint (i.e. CO2). The data transmission is two types of methods (1) Communication-Based (2) Component-Based. Here this paper concludes the compared study of component-based energy consumption using the Bellman-Ford Algorithm and Dijkstra's Algorithm. The results and finding measurement as discussed in this paper.

Legacy Data Center Network (DCN) Reviewed: considering Big Data Stream Mobile Computing (BDSMC) applications

Recently, Enterprises that operate over vast geographical areas uses multiple data centers to collect, store and process data in real time via energy-efficient acquisition, wirelessly transport clients or users data to the cloud. This paper seeks to technically review existing Data Center Networks (DCN) considering BDSMC applications. In this research, related works on Distributed Data Center Networks will be presented. Within the stream computing ecosystems, there are various network models but the pool of possible DCN topologies/architectures to adopt appears little and unfit for the purpose of BDSMC optimization. However, most related Data center architecture will be reviewed. Investigate efforts on both server centric and switch-centric models adaptable to BDSMC network layer. The extent of work done on distributed spine-leaf re-designed server-centric network construction so as to automatically harvest network interconnection into a ‘stellar’ dual-port server-centric SG network; how classical graph-based interconnection network translate network performance similar to generic works for BDSMC ecosystems. Review stellar transformation using the well-studied generalized hypercube family of interconnection networks for BDSMC ecosystems. The literature was searched from the databases: IEEE Xplore Digital Library, Springer Link Digital Library, and Google Scholar, IET Digital Library, Frontiers Library, ACM Digital Library repositories resulting in 98 papers after several eliminations ranging from year 2000-2022. In conclusion, state-of-the-art dual-port server-centric DCNs (FiConn, DCell, DPillar), etc, while looking at possible architectures with excellent comparative performance for BDSMC ecosystems. Research gaps are revealed for further study. Keywords: Data, Center, Network, Data Stream, Mobile Computing and applications

Carbon Management of Multi-Datacenter Based On Spatio-Temporal Task Migration

With the massive deployment of geographically distributed data centers (DCs) and the demand surge for cloud services, their high energy consumption and carbon pollution are increasingly problematic. Therefore, mitigating the harmful effects of the carbon footprint of DCs has become a critical challenge. This paper takes the first step toward analyzing the complementary characteristics of global multi-regional renewable energy sources (RES). There is an opportunity to schedule workloads flexibly and track RES to reduce emissions. Integrating DCs and inter-DC network to form a refined emission model for the trade-off between emission cutting effects from scheduling and carbon costs of workload migration, we propose the spatio-temporal task migration mechanism to pursue low carbon in dual-dimension: This paper shifts intensive workloads to locations sufficient in RES at a coarse scale, and adjust the execution time of the workload in response to real-time RES fluctuations at a fine scale. Thus, the emission overage is shifted and offset by RES in a complementary manner; meanwhile, the acceptance of RES is enhanced. Finally, experiments with real-world data show that our method can optimally coordinate demand with RES and mitigate carbon pollution in geographically distributed DCs, and verify the performance and applicability with various-parameter scenarios.

A Survey on Machine Learning for Geo-Distributed Cloud Data Center Management

Cloud workloads today are typically managed in a distributed environment and processed across geographically distributed data centers. Cloud service providers have been distributing data centers globally to reduce operating costs while also improving quality of service by using intelligent workload and resource management strategies. Such large scale and complex orchestration of software workload and hardware resources remains a difficult problem to solve efficiently. Researchers and practitioners have been trying to address this problem by proposing a variety of cloud management techniques. Mathematical optimization techniques have historically been used to address cloud management issues. But these techniques are difficult to scale to geo-distributed problem sizes and have limited applicability in dynamic heterogeneous system environments, forcing cloud service providers to explore intelligent data-driven and Machine Learning (ML) based alternatives. The characterization, prediction, control, and optimization of complex, heterogeneous, and ever-changing distributed cloud resources and workloads employing ML methodologies have received much attention in recent years. In this article, we review the state-of-the-art ML techniques for the cloud data center management problem. We examine the challenges and the issues in current research focused on ML for cloud management and explore strategies for addressing these issues. We also discuss advantages and disadvantages of ML techniques presented in the recent literature and make recommendations for future research directions.

A Framework for Several Electricity Retailers Cooperatively Implement Demand Response to Distributed Data Center

A distributed data center cluster (DDCC) consists of data centers that are geographically dispersed while managed by the same company. A DDCC is composed of several data centers in different regions, so it purchases electricity from several electricity retailers in different regions. Commonly, there is a lack of cooperation between the several electricity retailers. In this paper, a cooperation framework where several electricity retailers cooperatively implement incentive-based demand response (DR) to the same DDCC is proposed. Firstly, several electricity retailers which power for the same DDCC together publish their DR instructions to the DDCC. The DR instructions are analyzed in this paper. Then a novel DDCC energy management model is proposed for the DDCC to participate in the collaborative DR. Finally, a reasonable profit distribution mechanism is adopted for the several electricity retailers and the DDCC to allocate the entire profit of the collaborative DR. The case study verifies that the proposed cooperation framework can maximize the whole benefit of the DDCC and the several electricity retailers meanwhile ensures that the profits obtained by each member (the DDCC and the several electricity retailers) from the collaborative DR are not fewer than that of the independent operation.

MAGNet: Machine Learning Guided Application-Aware Networking for Data Centers

Modern data centers are witnessing fast-growing east-west traffic on their network infrastructure due to the highly distributed data center applications. Motivated by the heterogeneity of such application workloads, we propose in this paper an extensible network management architecture called MAGNET which enables application-aware intra-data center networking. The crux of MAGNET is the smart endpoint residing within end-hosts, which is empowered by machine learning combined with lightweight workload tracing to detect workload identities and enable workload-dependent packet tagging. The centralized management plane interface of MAGNET allows network functions to interpret packet tags and perform application-aware packet processing. We demonstrate the feasibility of the architecture via prototype implementation and extensive use case evaluation. Our experiments show that the smart endpoint can fingerprint many real-world applications with 99% accuracy only at 12% additional CPU, and that application-aware data plane can potentially bring substantial benefits in terms of security (e.g., via identity-based microsegmentation), CPU usage (e.g., for intrusion detection) and network latency (e.g., via TCP stack customization).

Performance Evaluation of Topologies for Multi-Domain Software-Defined Networking

Software-defined networking (SDN) is widely used in multiple types of data center networks, and these distributed data center networks can be integrated into a multi-domain SDN by utilizing multiple controllers. However, the network topology of each control domain of SDN will affect the performance of the multi-domain network, so performance evaluation is required before the deployment of the multi-domain SDN. Besides, there is a high cost to build real multi-domain SDN networks with different topologies, so it is necessary to use simulation testing methods to evaluate the topological performance of the multi-domain SDN network. As there is a lack of existing methods to construct a multi-domain SDN simulation network for the tool to evaluate the topological performance automatically, this paper proposes an automated multi-domain SDN topology performance evaluation framework, which supports multiple types of SDN network topologies in cooperating to construct a multi-domain SDN network. The framework integrates existing single-domain SDN simulation tools with network performance testing tools to realize automated performance evaluation of multi-domain SDN network topologies. We designed and implemented a Mininet-based simulation tool that can connect multiple controllers and run user-specified topologies in multiple SDN control domains to build and test multi-domain SDN networks faster. Then, we used the tool to perform performance tests on various data center network topologies in single-domain and multi-domain SDN simulation environments. Test results show that Space Shuffle has the most stable performance in a single-domain environment, and Fat-tree has the best performance in a multi-domain environment. Also, this tool has the characteristics of simplicity and stability, which can meet the needs of multi-domain SDN topology performance evaluation.

SPsync: Lightweight multi-terminal big spatiotemporal data synchronization solution

With the rapid increase in mobile device usage and the popularity of 5G communication technology, rich Big spatiotemporal Data has become an important research resource. How to quickly synchronize the Big spatiotemporal Data collected by various types of devices to multiple distributed data centers has become the key to further utilizing spatio-temporal data. Therefore, in this paper, a new lightweight Big Spatiotemporal Data synchronization scheme called SPsync, based on a traditional file synchronization algorithm, is proposed. The contributions of SPsync are as follows. (1) SPsync leads to the optimization of the file processing strategy, changing the algorithm from a serial algorithm to a parallel one. (2) Combined with Spark, a distributed Big Data processing framework is advanced to achieve optimization of distributed tasks and further improvement of algorithm performance. Further, we demonstrate the excellent performance of SPsync through rich experiments using multiple simulated and real datasets. Compared to the best incremental synchronization algorithms currently available, the synchronization speedup is 30% faster and CPU usage 20% lower. The algorithm itself is 20% faster and CPU usage reduced by more than 25% in a multi-node scenario.

MapReduce Task Scheduling in Heterogeneous Geo-Distributed Data Centers

Different data transmission times, processing times which are difficult to predict and node-dependent access times make MapReduce task scheduling rather complex. In this article, we consider the problem of scheduling MapReduce tasks to heterogeneous geo-distributed data centers to minimize the total tardiness. A new architecture is constructed to analyze data in the considered scenario. We model distinct data transmission levels, inter- and intra- data centers and heterogeneity of nodes mathematically. An algorithm framework is proposed to schedule MapReduce tasks to heterogeneous nodes in geographically distributed data centers. The proposed algorithm is suitable for both Hadoop MRv1 and MRv2. In terms of the number of idle containers detected in each heartbeat, the same number of tasks are selected from a sorted job sequence. For the map and reduce phases, two measurements are developed with data locality and completion time, respectively, based on which the classical Hungarian algorithm is adopted to optimally assign selected tasks to corresponding idle containers. Components and parameters of the proposal are statistically calibrated over a large set of random instances. A comparison of the proposed algorithm to existing methods for similar problems is carried out. Experimental results demonstrate the proposal is effective for the considered problem.