Replica Placement Algorithm Research Articles

TS-REPLICA: A novel replica placement algorithm based on the entropy weight TOPSIS method in spark for multimedia data analysis

Performance optimization based on node attributes is of profound significance in the replica placement algorithm used in Hadoop distributed file system (HDFS). Currently, most researchers studying replica placement algorithms consider only a single attribute or a few multiple attributes. However, a single attribute cannot accurately express the performance of a node. Therefore, this paper proposes a replica placement algorithm based on the entropy weight TOPSIS (technique for order preference by similarity to ideal solution) method, called TS-REPLICA. First, a multi-attribute matrix that comprehensively reflects the performance and load of nodes is defined. Then, a TOPSIS-based algorithm is proposed to calculate the performance score of each data node. In addition, the entropy weight method is introduced to derive the weights of attributes for balancing the influence of weights of multiple attributes. Next, the comprehensive load score of each data node in the Spark cluster, the average comprehensive load score of each rack, and the average comprehensive load score of the entire cluster are calculated, and replica placement is performed based on the obtained scores. Finally, the effectiveness of the proposed algorithm is verified on various datasets and test cases. The experimental results show that the TS-REPLICA algorithm outperforms the better comparison algorithm in execution number in Spark cluster.

Efficient Replica Migration Scheme for Distributed Cloud Storage Systems

With the wide adoption of large-scale internet services and big data, the cloud has become the ideal environment to satisfy the ever-growing storage demand. In this context, data replication has been touted as the ultimate solution to improve data availability and reduce access time. However, replica management systems usually need to migrate and create a large number of data replicas over time between and within data centers, incurring a large overhead in terms of network load and availability. In this paper, we propose CRANE, an effiCient Replica migrAtion scheme for distributed cloud Storage systEms. CRANE complements any replica placement algorithm by efficiently managing replica creation in geo-distributed infrastructures in order to (1) minimize the time needed to copy the data to the new replica location, (2) avoid network congestion, and (3) ensure the minimum desired availability for the data. Through simulation and experimental results, we show that CRANE provides a sub-optimal solution for the replica migration problem with lower computational complexity than its integer linear program formulation. We also show that, compared to OpenStack Swift, CRANE is able to reduce by up to 60 percent the replica creation and migration time and by up to 50 percent the inter-data center network traffic while ensuring the minimum required data availability.

A novel replica placement algorithm for minimising communication cost in distributed storage platform

In large-scale distributed systems, replication service has been playing a critical role to improve the availability and reliability of user data. Conventionally, the existing replication services mainly concentrate on how many replicas are needed to maintain desirable availability and reliability rather than how to place replicas on the most suitable storage nodes. As a result, the communication-related costs when accessing data are significantly increased, which in turn degrades the execution performance of user applications. In this paper, we propose a novel replica placement algorithm which is designed to minimise the communication cost when accessing or managing replicas in a large-scale storage platform. In the proposed algorithm, the replica placement problem is formulised a classical multi-knapsack problem, and two heuristic metrics are introduced to obtain the sub-optimal solution of this problem. A lot of experiments are conducted to investigate the performance of the proposed algorithm. The experimental results indicate that our replica placement algorithm outperforms many existing approaches in terms of different performance metrics. In addition, the proposed algorithm can also significantly improve the execution efficiency for data-intensive applications, which are very common in nowadays large-scale distributed systems, such as grid and cloud.

A novel replica placement algorithm for minimising communication cost in distributed storage platform

In large-scale distributed systems, replication service has been playing a critical role to improve the availability and reliability of user data. Conventionally, the existing replication services mainly concentrate on how many replicas are needed to maintain desirable availability and reliability rather than how to place replicas on the most suitable storage nodes. As a result, the communication-related costs when accessing data are significantly increased, which in turn degrades the execution performance of user applications. In this paper, we propose a novel replica placement algorithm which is designed to minimise the communication cost when accessing or managing replicas in a large-scale storage platform. In the proposed algorithm, the replica placement problem is formulised a classical multi-knapsack problem, and two heuristic metrics are introduced to obtain the sub-optimal solution of this problem. A lot of experiments are conducted to investigate the performance of the proposed algorithm. The experimental results indicate that our replica placement algorithm outperforms many existing approaches in terms of different performance metrics. In addition, the proposed algorithm can also significantly improve the execution efficiency for data-intensive applications, which are very common in nowadays large-scale distributed systems, such as grid and cloud.

An Incentive Compatible Mechanism for Replica Placement in Peer-Assisted Content Distribution

Content delivery is a key technology on the Internet to achieve large scale, low-latency, reliable, and intelligent data delivery. Replica placement (RP) is a key machinery in content delivery systems to achieve efficient and effective content delivery. This work proposes a novel decentralized algorithm for the replica placement in peer-assisted content delivery networks with simultaneous considerations for peer incentives. By applying techniques from the algorithmic mechanism design theory, the authors show the incentive compatibility of the proposed algorithm. Experiments were conducted to validate the properties of the proposed method and comparisons were made with the state-of-the-art RP algorithms.

Choosing the best Bin Packing Algorithm for Replica Placement in Multi-Tenant Cloud System

Of late, Cloud Computing is visibly seen to reduce infrastructure costs with high data availability and performance conforming to service level agreement for both the service providers and the users. With the rapid and explosive growth of the number of cloud users, Cloud Data Management System must serve an array of different analytical and transactional workloads. Hence, to ensure the scalability in a multi-tenant system, replica placement algorithms always come into the picture appropriately. In our work, we have vividly analyzed various replica placement algorithms in terms of their performance and tried to find the beneficial aspects to be the fittest one to tackle the situation when the actual observed workloads are immensely deviated from the estimated workloads.

Flexible replica placement for enhancing the availability in edge computing environment

As edge devices that make up the Internet become more powerful, these devices in edge computing environment will even perform better in improving the computing and storage capabilities of system. In this paper, to enhancing data availability and improve cloud storage performance in edge computing environment, an adaptive replica placement strategy in edge computing environment is proposed to study the relationship among user access characteristics, the number of replicas, and the replicas location. To deal with the volatility of user access, the strategy in this paper takes data block as the data unit and the dynamic replica creation algorithm (DRC-GM) is proposed. DRC-GM uses the relationship between access frequency of data blocks and the number of replicas to dynamically adjust the number of replicas to ensure data availability requirements. To find the optimal location for replica distributed placement, in this paper, the replica placement algorithm (RP-FNSG) is proposed. And replica number, node performance and replica consistency are considered. Experiment results show that DRC-GM and RP-FNSG algorithms in edge computing environment can greatly improve system performance in terms of better prediction accuracy, shorter access response, higher effective network and storage space usage, which enhancing the data availability.

Adaptive Avatar Handoff in the Cloudlet Network

In a traditional big data network, data streams generated by User Equipments (UEs) are uploaded to the remote cloud (for further processing) via the Internet. However, moving a huge amount of data via the Internet may lead to a long End-to-End (E2E) delay between a UE and its computing resources (in the remote cloud) as well as severe traffic jams in the Internet. To overcome this drawback, we propose a cloudlet network to bring the computing and storage resources from the cloud to the mobile edge. Each base station is attached to one cloudlet and each UE is associated with its Avatar in the cloudlet to process its data locally. Thus, the E2E delay between a UE and its computing resources in its Avatars is reduced as compared to that in the traditional big data network. However, in order to maintain the low E2E delay when UEs roam away, it is necessary to hand off Avatars accordingly-it is not practical to hand off the Avatars' virtual disks during roaming as this will incur unbearable migration time and network congestion. We propose the LatEncy Aware Replica placemeNt (LEARN) algorithm to place a number of replicas of each Avatar's virtual disk into suitable cloudlets. Thus, the Avatar can be handed off among its cloudlets (which contain one of its replicas) without migrating its virtual disk. Simulations demonstrate that LEARN reduces the average E2E delay. Meanwhile, by considering the capacity limitation of each cloudlet, we propose the LatEncy aware Avatar hanDoff (LEAD) algorithm to place UEs' Avatars among the cloudlets such that the average E2E delay is minimized. Simulations demonstrate that LEAD maintains the low average E2E delay.

Energy-efficient fault-tolerant replica management policy with deadline and budget constraints in edge-cloud environment

With the development of large-scale distributed systems such as grids and clouds, data replication management has become a hot research topic. Although replica management can improve the cluster system performance, it also brings a series of management and overhead issues. Therefore, the energy-efficient fault-tolerant replica management policy with the deadline and budget constraints in the edge-cloud environment is proposed. The experiments show that the proposed dynamic replica placement algorithm can effectively reduce the mean job time, reduce the use of network bandwidth and improve the utilization of storage space. Considering the issue of energy efficiency, the energy-aware cluster scaling strategy is proposed to reduce system energy consumption and achieve energy efficiency by sleeping and waking up the data nodes according to the load state of the system. Besides, in order to avoid access failures and data loss caused by node failures, the node failure recovery method based on availability metrics is used to deal with node failures. The experiments show that the performance of the proposed algorithm is better than the other algorithms in terms of energy efficiency and fault tolerance.

Leveraging A Multi-Objective Approach to Data Replication in Cloud Computing Environment to Support Big Data Applications

Increased data availability and high data access performance are of utmost importance in a large-scale distributed system such as data cloud. To address these issues data can be replicated in various locations in the system where applications are executed. Replication not only improves data availability and access latency but also improves system load balancing. While data replication in distributed cloud storage is addressed in the literature, majority of the current techniques do not consider different costs and benefits of replication from a comprehensive perspective. In this paper, we investigate replica management problem (which is formulated using dynamic programming) in cloud computing environments to support big data applications. To this end, we propose a new highly distributed replica placement algorithm that provides cost-effective replication of huge amount of geographically distributed data into the cloud to meet the quality of service (QoS) requirements of data-intensive (big data) applications while ensuring that the workload among the replica data centers is balanced. In addition, the algorithm takes into account the consistency among replicas due to update propagation. Thus, we build up a multi-objective optimization approach for replica management in cloud that seeks near optimal solution by balancing the trade-offs among the stated issues. For verifying the effectiveness of the algorithm, we evaluated the performance of the algorithm and compared it with two baseline approaches from the literature. The evaluation results demonstrate the usefulness and superiority of the presented algorithm for conditions of interest.

Dataset replica placement strategy under a response time constraint in the cloud

In cloud computing environment, especially data-intensive systems, large amounts of datasets are stored in distributed data centres, and are often retrieved by users in different regions. To reduce the users' response time, replicating the popular datasets to multiple suitable data centres is an advisable choice, as tasks can access the datasets from a nearby site. Nevertheless, the dataset replicas' suitable storage placement selection is still an important issue that should be solved urgently from the response time constraint view, for the reason that too many replicas are infeasible in practice. In this paper, we first propose a comprehensive dataset response time estimation model, then present a replica placement model based on Steiner tree. After that, an approximate replica placement algorithm under a response time constraint in the cloud is given using Kruskal minimum spanning tree. At last, a practical and reasonable performance evaluation is designed and implemented. Both the theoretical analysis and simulations conducted on general (random) datasets show the efficiency and effectiveness of the proposed strategy in the cloud.

QoS Aware Replica Control Strategies for Distributed Real Time Database Management System

In distributed real time database management system (DRT-DBMS), QoS aware replication control strategy is necessary to ensure availability and to improve the system performance. Since user request and need of data is dynamic in nature, it requires an approach which addresses the problem of data distribution and also resolves them in DRT-DBMS. A large quantity of data file is produced so we need a dynamic replication approach through which we can reduce the data access time. In order to replicate it, at the nearest site, replica placement and replacement algorithms are required to be evaluated. This paper presents a heuristic approach Greedy-Cover Firefly algorithm in which replica is placed dynamically based on QoS requirement and replaced with the optimal adaptive replica replacement algorithm for each data object. The replication cost required for copying data object to each set of a node is calculated to select the optimal node. Here, the replication decision is taken on the basis of access history of data on multiple sites. Due to the limited storage capacity of each node, the value of each file is calculated, and the file with the smallest value is removed to replace other files. A detailed simulation shows that the proposed algorithms can greatly improve the system performance to reduce the system resources consumed with respect to time and QoS requirement.

Framework for replica placement over cooperative edge networks

Multimedia traffic is increasing sharply as an important component of Internet traffic to consume the bandwidth with various applications. Although most existing studies on edge networks focus on guaranteeing the quality of service for users by decreasing the latency and cost, the volume of traffic has become notably large, and the state of the network is unstable. The cloud platform and Software-Defined Network (SDN), as new technologies, undertake several functions from the edge networks such that the edge networks can concentrate on the delivery part and make optimal decisions. In this paper, we propose Cloud Co-CDNs (C3), which a prototype that transfers the management part of edge networks to the cloud with the software-defined network, and cooperation among providers of edge networks is considered in the prototype. Next, we design a stochastic model for describing the workflow in the C3 and develop a heuristic algorithm for replica placement to trade off among the latencies that are produced by the delivery process, the updating process and the replication process. Finally, we evaluate the performances of the proposed algorithms through experimental simulations. Simulation results demonstrate that the proposed algorithms are promising.

Availability Is Not Enough: Minimizing Joint Response Time in Peer-Assisted Cloud Storage Systems

Cloud storage systems have been rapidly emerging recently since they bring fast and convenient storage services to users. We note that providing those service in a private-cloud way is costly, i.e., energy, bandwidth, and storage hardware are expensive for maintaining a cloud storage platform. To cope with the aforementioned issues, our idea is to exploit unused space and bandwidth at user-site machines to build a peer-assisted cloud storage platform. That is, in a peer-assisted cloud storage system, each peer helps back up data of other peers, and all peers form a virtual storage system. However, due to the uneven data popularity and the heterogeneous user upload capacity in peer-to-peer networks, bandwidth-oblivious replica placement might cause extremely long response time for certain data objects. Thus, this paper proposes a new metric called joint response time, which not only considers the waiting time when the requested data are unavailable but also the queuing delay and service time when data become available. We then design a 2-D Markov model to estimate this metric and propose a bandwidth-aware ( BA) replica placement algorithm to reduce the joint response time. Our trace-driven evaluation results validate that the proposed BA algorithm can efficiently utilize the upload capacity of each peer to provide peers a shorter joint response time.

Optimized Data Replication for Small Files in Cloud Storage Systems

Cloud storage has become an important part of a cloud system nowadays. Most current cloud storage systems perform well for large files but they cannot manage small file storage appropriately. With the development of cloud services, more and more small files are emerging. Therefore, we propose an optimized data replication approach for small files in cloud storage systems. A small file merging algorithm and a block replica placement algorithm are involved in this approach. Small files are classified into four types according to their access frequencies. A number of small files will be merged into the same block based on which type they belong to. And the replica placement algorithm helps to improve the access efficiencies of small files in a cloud system. Related experiment results demonstrate that our proposed approach can effectively shorten the time spent reading and writing small files, and it performs better than the other two already known data replication algorithms: HAR and SequenceFile.

On optimally reducing search costs for placing replicas in Symphony P2P networks

Peer-to-peer (P2P) overlay networks are widely employed in distributed systems. Furthermore, the number of hops required by a node for searching an object is the fundamental search cost of a P2P network. Hence, since placing replicas can efficiently reduce such a cost of the whole system, how to deploy replicas to reduce it as much as possible becomes a critical problem of P2P networks. In this study, the authors investigate this centre problem for the randomised P2P network ‘Symphony’. Particularly, the authors present a stochastic analysis on optimal replica placements, in terms of number of total hops, as well as proposing an optimal replica placement algorithm to perform a simulation study on this issue. Both the theoretical and experimental results show that locations of optimal replica placements in Symphony are not necessarily closest to the target node in the identifier space. Hence, based on the results, the authors evaluate one existing simple but experimentally efficient replica placement strategy, which exactly allocates replicas at nodes closest to the target node, to provide some guidelines on designing more efficient strategy applicable to the real-world Symphony.

Research on the Key Technologies of Resource Dynamic Management in Cloud Computing Environment

For the "hot spots" problem caused by the number of replica could not satisfy the soaring request from users, we present a Cloud Computing storage resource replica dynamic scheduling scheme based on HDFS. There are two sub algorithms which is the number of replica dynamic adjustment algorithm and replica placement algorithm. This scheme could improve system stability and file access through put. The experiment result shows this optimized scheme lower the access delay obviously than the original scheme with some cost of storage resource and network workload. For the problem of metadata management, this paper introduced a distributed metadata management scheme. This scheme distributed metadata into several name node according to their ability, and provides an efficient inquiry algorithm for metadata to improve system availability and reliability. The experiment result shows the multi-name nodes system with this metadata management scheme could improve the performance than the original system facing the huge metadata operation requests.

A cloud middleware for assuring performance and high availability of soft real-time applications

Applications are increasingly being deployed in the cloud due to benefits stemming from economy of scale, scalability, flexibility and utility-based pricing model. Although most cloud-based applications have hitherto been enterprise-style, there is an emerging need for hosting real-time streaming applications in the cloud that demand both high availability and low latency. Contemporary cloud computing research has seldom focused on solutions that provide both high availability and real-time assurance to these applications in a way that also optimizes resource consumption in data centers, which is a key consideration for cloud providers. This paper makes three contributions to address this dual challenge. First, it describes an architecture for a fault-tolerant framework that can be used to automatically deploy replicas of virtual machines in data centers in a way that optimizes resources while assuring availability and responsiveness. Second, it describes the design of a pluggable framework within the fault-tolerant architecture that enables plugging in different placement algorithms for VM replica deployment. Third, it illustrates the design of a framework for real-time dissemination of resource utilization information using a real-time publish/subscribe framework, which is required by the replica selection and placement framework. Experimental results using a case study that involves a specific replica placement algorithm are presented to evaluate the effectiveness of our architecture.

Dynamic replica placement and selection strategies in data grids— A comprehensive survey

Data replication techniques are used in data grid to reduce makespan, storage consumption, access latency and network bandwidth. Data replication enhances data availability and thereby increases the system reliability. There are two steps involved in data replication, namely, replica placement and replica selection. Replica placement involves identifying the best possible node to duplicate data based on network latency and user request. Replica selection involves selecting the best replica location to access the data for job execution in the data grid. Various replica placement and selection algorithms are available in the literature. These algorithms measure and analyze different parameters such as bandwidth consumption, access cost, scalability, execution time, storage consumption and makespan. In this paper, various replica placement and selection strategies along with their merits and demerits are discussed. This paper also analyses the performance of various strategies with respect to the parameters mentioned above. In particular, this paper focuses on the dynamic replica placement and selection strategies in the data grid environment.

Distributed replica placement algorithms for correlated data

In distributed systems, data may be correlated due to accesses from clients and the correlation has some impact on date placement, and existing research works focus on independent data objects. In this paper, we address both the scalability and the stability of the data placement solutions in internet environment. We first show that replica allocation decisions can be made locally for each replica site in a tree network, with data access knowledge of its neighbors. We then develop a new replication cost model for correlated data objects in Internet environment. Based on the cost model and the algorithms in previous research, we develop a distributed optimal replica allocation algorithm (DOPR) for correlated data in internet environment. A distributed heuristic algorithm (DHPR) is then developed to efficiently make replica placement decisions. The algorithm obtains sub-optimal solutions for the correlated data model and yields significant performance gains. Experimental studies show that the distributed heuristic allocation algorithm significantly outperforms the general frequency-based replication schemes (in which the replication decision of each data object is made based on the number of accesses on that data object).