Replica Management Research Articles

An Experimental Evaluation of Performance of a Hadoop Cluster on Replica Management

Hadoop is an open source implementation of the MapReduce Framework in the realm of distributed processing. A Hadoop cluster is a unique type of computational cluster designed for storing and analyzing large datasets across cluster of workstations. To handle massive scale data, Hadoop exploits the Hadoop Distributed File System termed as HDFS. The HDFS similar to most distributed file systems share a familiar problem on data sharing and availability among compute nodes, often which leads to decrease in performance. This paper is an experimental evaluation of Hadoop's computing performance which is made by designing a rack aware cluster that utilizes the Hadoop’s default block placement policy to improve data availability. Additionally, an adaptive data replication scheme that relies on access count prediction using Langrange’s interpolation is adapted to fit the scenario. To prove, experiments were conducted on a rack aware cluster setup which significantly reduced the task completion time, but once the volume of the data being processed increases there is a considerable cutback in computational speeds due to update cost. Further the threshold level for balance between the update cost and replication factor is identified and presented graphically.

DRMST: A Novel Dynamic Replica Management Strategy Based on TOPSIS in Cloud Storage

DRMST: A Novel Dynamic Replica Management Strategy Based on TOPSIS in Cloud Storage

Research on Resource Scheduling in Cloud Computing: Issues and Solutions

.Resource scheduling is the core technology to provide efficient and reliable services in cloud computing, and it is the basis of cloud computing to implement quick deploy and rapid response and save money. This article firstly introduces the research status of the resource scheduling in cloud computing including resource scheduling policies, replica technology and metadata management. Next we analyze the issues of Hadoop platform in resource scheduling including high latency, small files I/O, single point of failure and hot data. On the basis of these, the effective resource scheduling and management mechanisms are given including dynamic replica management, metadata management and horizontal scalability.

ATLAS Replica Management in Rucio: Replication Rules and Subscriptions

The ATLAS Distributed Data Management system stores more than 150PB of physics data across 120 sites globally. To cope with the anticipated ATLAS workload of the coming decade, Rucio, the next-generation data management system has been developed. Replica management, as one of the key aspects of the system, has to satisfy critical performance requirements in order to keep pace with the experiment's high rate of continual data generation. The challenge lies in meeting these performance objectives while still giving the system users and applications a powerful toolkit to control their data workflows. In this work we present the concept, design and implementation of the replica management in Rucio. We will specifically introduce the workflows behind replication rules, their formal language definition, weighting and site selection. Furthermore we will present the subscription component, which offers functionality for users to proclaim interest in data that has not been created yet. This contribution describes the concept and the architecture behind those components and will show the benefits made by this system.

Integrated Obj_FedRep: Evaluation of Surrogate Object based Mobile Cloud System for Federation, Replica and Data Management

Advancement in wireless and mobile communication with cloud has the potential to revolutionize computing by the illusion of a virtually infinite computing infrastructure, namely, the mobile cloud. However, mobile cloud faces many challenges such as dependency on continuous network connections, data sharing applications and federation with multiple service providers. In this paper a novel technique called Integrated Obj_FedRep (IOFR) is proposed, which federates two or more service providers for the purpose of load balancing and cloud replication for fast and easy access. The proposed IOFR model creates a real-time ongoing clone of database servers of different service providers using surrogate object in the mobile support station and an exact copy updated with the database server in synchronized and unsynchronized manner. The model handles the federation and replica at the object level which in turn provides better response time and minimizes the overall network traffic incurred due to mobility and database server failure. An extensive simulation of the IOFR has been done and it shows that the proposed technique improves the response times, achieves better bandwidth utilization and provides support for disconnection.

Exploring the policy selection of the P2P VoD system: A simulation-based research

The P2P-assisted video-on-demand (P2P VoD) service has achieved tremendous success among the Internet users. There are three core strategies in the P2P VoD system: the piece selection policy, the peer selection policy as well as the replica management policy. Different from the existing research works that only consider single policy optimization, we for the first time study the existing P2P VoD policies by using a simulation framework to understand the performance of different policy compositions. The simulation results indicate that when the bandwidth and storage resources are limited in the P2P VoD system, the composition of the sequential piece selection policy, the cascading peer selection policy and the proportional replica management policy has the best performance among all different policy compositions. However, when the bandwidth and storage resources are sufficient in the P2P VoD system, there will be little difference between different choices.

MORM: A Multi-objective Optimized Replication Management strategy for cloud storage cluster

Effective data management is an important issue for a large-scale distributed environment such as data cloud. This can be achieved by using file replication, which efficiently reduces file service time and access latency, increases file availability and improves system load balancing. However, replication entails various costs such as storage and energy consumption for holding replicas. This article proposes a multi-objective offline optimization approach for replica management, in which we view the various factors influencing replication decisions such as mean file unavailability, mean service time, load variance, energy consumption and mean access latency as five objectives. It makes decisions of replication factor and replication layout with an improved artificial immune algorithm that evolves a set of solution candidates through clone, mutation and selection processes. The proposed algorithm named Multi-objective Optimized Replication Management (MORM) seeks the near optimal solutions by balancing the trade-offs among the five optimization objectives. The article reports a series of experiments that show the effectiveness of the MORM. Experimental results conclusively demonstrate that our MORM is energy effective and outperforms default replication management of HDFS (Hadoop Distributed File System) and MOE (Multi-objective Evolutionary) algorithm in terms of performance and load balancing for large-scale cloud storage cluster.

Research and Improvement of the Hot Small File Storage Performance under HDFS

In order to deal with a large number of small files and hotspot data program in Hadoop distributed file system (HDFS)[1,, according to the exit proposal, this paper proposes a new the hotspot data processing model. The model proposals to change the block size, the introduction of efficient indexing mechanism to improve the dynamic replica management strategy and design of the new HDFS architecture to save space, speed up system processing, and enhance security.

A three-phase energy-saving strategy for cloud storage systems

In the running process of cloud data center, the idle data nodes will generate a large amount of unnecessary energy consumption. Furthermore, the resource misallocation will also cause a great waste of energy. This paper proposes a three-phase energy-saving strategy named TPES in order to save energy and operational costs for cloud suppliers. The three phases are replica management based on variable replication factor, cluster reconfiguration according to the optimal total costs and state transition based on observed and predicted workloads. These three phases save energy for the system at different levels which enhance the adaptability of our strategy. We evaluate our strategy using the expanded CloudSim toolkit and the results show that the proposed strategy achieves better energy reduction under different conditions in comparison with the existing schemes.

Review of Data Replication Techniques for Mobile Computing Environment

With the tremendous growth of the portable computing devices and wireless communication, the mobile computing has become extremely popular. The wireless enabled portable computing devices with massive storage capacity and high-end processing capabilities have started to make the extensive use of mobile databases already. The rising popularity in these computing paradigms demands that the mobile computing be reliable enough to ensure the continuous data availability and integrity. However mobility comes at the cost of bandwidth, limited power, security and interference. In modern mobile computing systems data replication has been adopted as an efficient means to ensure the data availability and integrity as well as an effective means to achieve the fault tolerance. Data replication not only ensure the availability of the data but also reduce the communication cost, increase data sharing and increase the safety of the critical data. Furthermore, it also determine when and where (location) to place the replica of data, controlling the number of data replicas over a network for efficient utilization of the network resources. In this study we survey the research work in data replication for mobile computing. We reviewed some of the existing data replication techniques proposed by the research community for mobile computing environment for efficient management of data replicas.

Replica creation strategy based on quantum evolutionary algorithm in data gird

As a research branch of grid computing, data grid focuses on the management of large-scale distributed data sets. Replica management is one of the most important issues in the data grid, which can offer fast data access time, high data availability and low bandwidth consumption. Computing Intelligent Algorithm (CIA) has been proved to be effective in the solution of large-scale distributed computing problems, whereas Quantum Evolutionary Algorithm (QEA) is one of these excellent optimization algorithms and little literatures are made for its application in Data Grid Replica Management (DGRM). This paper focuses on the application of the QEA in data grid replica creation strategy. A QEA-based global replica creation strategy is proposed after reviewing the replica creation strategies. The optimization model is divided into single and multi data replica creation two parts. The representation, evaluation and constraint procedure three key technologies problems for each part are discussed in detail. The detail algorithm of QEA based replica creation is provided. The experiments were carried out with OptorSim, and the results have shown that QEA-based replica creation strategy can effectively reduce the job response time and network bandwidth consumption, comparing to Genetic Algorithms (GAs), Ant Colony Optimization (ACO), Particle Swarm Optimization (PSO) algorithms. Especially, its performance becomes better and better with the incensement of the number of jobs. The non-parametric statistical tests are used to verify the significant of QEA.

Research on the Solution to Grid Data Replication Management in Distributed Environment

Data grid replica management is a hot topic in the field of grid computing. The key technologies to the grid data replica management solution in distributed environment include: replica creation, replica deletion, replica location and selection, data access pattern, replica consistency management and replica security management. For the reasons above, based on current replication strategy, according to the characteristics of applications and the requirements of users, it is necessary to design comprehensively and implement data replication strategy and technology that conforms to data grid characteristics. In the article, the authors discuss the contents studied in grid data replication technology in depth, and analyze each data replication key technology in data grid system. In the aspect of data access, it breaks through traditional mode and adopts distributed file sharing mode to data access. The experiments show that its performance is improved than traditional download mode. It reduces the response time of the data and improves the performance of job-execution.

A dynamic replica management strategy in data grid

Data Grid provides scalable infrastructure for storage resource and data files management, which supports several large scale applications. Due to limitation of available resources in grid, efficient use of the grid resources becomes an important challenge. Replication is a technique used in data grid to improve fault tolerance and to reduce the bandwidth consumption. This paper proposes a Dynamic Hierarchical Replication (DHR) algorithm that places replicas in appropriate sites i.e. best site that has the highest number of access for that particular replica. It also minimizes access latency by selecting the best replica when various sites hold replicas. The proposed replica selection strategy selects the best replica location for the users' running jobs by considering the replica requests that waiting in the storage and data transfer time. The simulated results with OptorSim, i.e. European Data Grid simulator show that DHR strategy gives better performance compared to the other algorithms and prevents unnecessary creation of replica which leads to efficient storage usage.

Management and Placement of Replicas in a Hierarchical Data Grid

In large systems, data management is only possible through the use of replication techniques. In order to improve availability and ensure fast and efficient access, we propose in this paper a dynamic replication algorithm which takes into account also the placement of data in a hierarchical data grid. Experimental results show the effectiveness of our approach in terms of response time and availability of data in the system.

Research of Replica Management Technology of Resource on Manufacturing Grid

The resources on Manufacturing Grid are dynamic, diverse, distributed and isomeric, so their management is significantly complicated. However, people always want to obtain the newest resources’ information in the recent distance, in the shortest time. Thus, replica management of resource becomes necessary. This paper makes the following contributions. First, we elaborate the resources’ encapsulation technology on Manufacturing Grid. Second, we describe the main idea of replica management technology. In addition, we explain the working principle of component that is management replica—RLS (Replica Location Service). Finally, we first bring forward replica management technology of resource on Manufacturing Grid, and design a solution that resources’ replica management technology on Manufacturing Grid.

VTR-based Replica Management Service in Data Grid

VTR-based Replica Management Service in Data Grid

QoS enhancements for global replication management in peer to peer networks

Replica Management is a key issue to reduce the bandwidth consumption, to improve data availability and to maintain data consistency in large distributed systems. Global Replica Management (GRM) means to maintain the data consistency across the entire network. It is preferable particularly for multi-group distributed systems. On the other hand, GRM is not favorable for many applications because a very large number of message passes is needed for replica management processes. In this paper, in order to reduce the number of message passes needed to achieve the efficient GRM strategy, an interconnection structure called the Distributed Spanning Tree (DST) has been employed. The application of DST converts the peer network into logical layered structures and thereby provides a hierarchical mechanism for replication management. It is proved that this hierarchical approach improves the data availability and consistency across the entire network. In addition to these, it is also proved that the proposed approach reduces the data latency and the required number of message passes for any specific application in the network.

Study on Replica Strategy in the Grid Environment

An effective replica management method can save bandwidth, reduce latency, balance load, and improve system reliability. A comprehensive scheme is introduced to manage replica for the grid environment. The schemes of replica creation and placement based on the frequency of visits and storage space, replica replacement based on the replica value, and replica selection based on comprehensive performance are proposed. Our scheme can achieve optimization of data distribution and replication, improve the efficiency of data visits in the grid environment.

Implementation of a distributed file storage with replica management in Peer-to-Peer environments

In this work, we make utilisation of users' computers involved in this distributed storage system easily, and users share their powerful resources with this Peer-to-Peer (P2P) environment. Users can download the resources from this P2P distributed storage system. To transfer more efficiently, this paper also proposes a parallel transfer algorithm for every node and replication mechanism. The experiment results have shown that it is an effective parallel transfer method, which not only uses network bandwidth more, but also reduces the download transfer time.

DYFRAM: dynamic fragmentation and replica management in distributed database systems

In distributed database systems, tables are frequently fragmented and replicated over a number of sites in order to reduce network communication costs. How to fragment, when to replicate and how to allocate the fragments to the sites are challenging problems that has previously been solved either by static fragmentation, replication and allocation, or based on a priori query analysis. Many emerging applications of distributed database systems generate very dynamic workloads with frequent changes in access patterns from different sites. In such contexts, continuous refragmentation and reallocation can significantly improve performance. In this paper we present DYFRAM, a decentralized approach for dynamic table fragmentation and allocation in distributed database systems based on observation of the access patterns of sites to tables. The approach performs fragmentation, replication, and reallocation based on recent access history, aiming at maximizing the number of local accesses compared to accesses from remote sites. We show through simulations and experiments on the DASCOSA distributed database system that the approach significantly reduces communication costs for typical access patterns, thus demonstrating the feasibility of our approach.