Dynamic Replication Research Articles

The ATLAS Data Management System Rucio: Supporting LHC Run-2 and beyond

With this contribution we present some recent developments made to Rucio, the data management system of the High-Energy Physics Experiment ATLAS. Already managing 300 Petabytes of both official and user data, Rucio has seen incremental improvements throughout LHC Run-2, and is currently laying the groundwork for HEP computing in the HL-LHC era. The focus of this contribution are (a) the automations that have been put in place such as data rebalancing or dynamic replication of user data, as well as their supporting infrastructures such as real-time networking metrics or transfer time predictions; (b) the flexible approach towards inclusion of heterogeneous storage systems, including object stores, while unifying the potential access paths using generally available tools and protocols; (c) machine learning approaches to help with transfer throughput estimation; and (d) the adoption of Rucio for two other experiments, AMS and Xenon1t. We conclude by presenting operational numbers and figures to quantify these improvements, and extrapolate the necessary changes and developments for future LHC runs.

HaDaap: A hotness‐aware data placement strategy for improving storage efficiency in heterogeneous Hadoop clusters

SummaryEnterprises increasingly use the Hadoop Distributed File System (HDFS) to manage and store big data for many applications. However, HDFS uses triple replication, leading to staggering data center storage costs. As big data increases in volume and its heat levels becomes more sensitive, there comes a point where storing so much cold data actually makes it less accessible and more expensive. Meanwhile, as data centers expand, the heterogeneity of nodes also becomes an issue. Rack‐aware data placement adopted by HDFS results in an unbalanced load and uneven resource allocation because it ignores the data nodes' heterogeneity. Here, we attempt to resolve these problems by proposing a hotness‐aware data placement strategy (named HaDaap). In HaDaap, the first step is to use a hotness‐aware data clustering algorithm to set the data's degree of heat. Then, cold data (with a redundancy of erasure code) are placed through a Double Sort Exchange algorithm to reduce storage costs and increase data availability. Finally, hot data are placed via a dynamic replication placement mechanism that comprehensively factors availability, load, and storage costs. Experimental results show that with these enhancements, HaDaap uses resources rationally and substantially reduces storage costs by considering the difference of data hotness in heterogeneous Hadoop clusters.

A Review of Static and Dynamic Data Replication Mechanisms for Distributed Systems

A Review of Static and Dynamic Data Replication Mechanisms for Distributed Systems

A new Prefetching-aware Data Replication to decrease access latency in cloud environment

Data replication is an effective technique that decreases retrieval time, thus reducing energy consumption in Cloud. When necessary files aren't locally available, they will be fetched from remote locations that is very high-time consuming process. Therefore, it is superior to pre-replicate the popular files. Even though few previous works considered prediction-based replication strategy, the prediction is not precise at many situations and occupies the storage. To address these challenges, a new dynamic replication strategy called Prefetching-aware Data Replication (PDR) is proposed, which determines the correlation of the data files using the file access history and pre-fetches the most popular files. So, the next time that this site requires a file, it will be locally available. In addition, due to the storage space restriction, replica replacement strategy plays a vital role. PDR strategy can ascertain the importance of valuable replicas based on the fuzzy inference system with four input parameters (i.e., number of accesses, cost of replica, the last time the replica was accessed, and data availability). Extensive experiments with CloudSim show that PDR achieves high data availability, high hit ratio, low storage and bandwidth consumption. On average PDR reduces over 35% of response time when compared to the other algorithms.

A Dynamic Replica Factor Calculator for Weighted Dynamic Replication Management in Cloud Storage Systems

Data replication is a widely-used technique in cloud storage systems to ensure availability of data. In static replication policies, the number of replicas to be created is decided statically at the time of cloud system setup. The replication of data needs to be dynamic considering changing pattern in user request and storage capacity. This will ensure that storage space is not wasted by keeping unnecessary replicas for least used data. In this work, a strategy is proposed to dynamically set the replica factor for each data item considering the popularity of data, its current replication factor and the number of active nodes present in the cloud storage. The proposed strategy is implemented in Hadoop Distributed File System (HDFS) and the experimental results shows that proposed method maintains an optimal number of replicas for each data item based on its popularity, without compromising on the availability constraint of cloud storage.

The Evolutionary Game of Electronic Seal Usage Behaviour Supervision From the Perspective of Credit and Penalty

The illegal use of electronic seals has the potential to harm both the contracting parties and the country’s economy. This paper studies the phylogenetic relationship between government regulatory bodies and enterprise behaviors through the payoff matrix and an evolutionary game model. According to both the equilibrium point and stability of the game’s dynamic replication relationship phase diagram, this paper analyzes the incentive effects of the legal use of management costs and the illegal use of economic penalties, government regulatory costs, and credit ratings on the use of electronic seals. The results show that the steady state does not exist as a time extension of government supervision and standardize the behaviors of electronic seal enterprises. In addition, the selection specification evolutionary path using electronic seals is different based on the different initial conditions. The use of the restricted speed and the evolutionary result not only depends on the choice of the enterprise’s standard initial probability of usage behavior but also on the initial probability of government departmental supervision. This occurs without using any means of supervision and penalty measures under the conditions that all enterprises tend to choose not to regulate the use of the electronic seal. Due to regulation with a low probability of no significant effect of the standard, the need to reach a certain level of supervision to regulate the use of electronic seals was improved. The government simply increasing the penalty is not an effective solution since the electronic seal does not regulate the problematic use of credit rating factors. On the one hand, it reduces the probability of government regulations, and on the other hand, it greatly improves the standard probability of the use of electronic seals. Finally, conclusions are based on the governmental point of view to promote the use of electronic seal specification recommendations.

Improving “tail” computations in a BOINC-based Desktop Grid

AbstractA regular Desktop Grid bag-of-tasks project can take a lot of time to complete computations. An important part of the process is tail computations: when the number of tasks to perform becomes less than the number of computing nodes. At this stage, a dynamic replication could be used to reduce the time needed to complete computations. In this paper, we propose a mathematical model and a strategy of dynamic replication at the tail stage. The results of the numerical experiments are given.

A dynamic replication management strategy in distributed GIS

Replication strategy is one of effective solutions to meet the requirement of service response time by preparing data in advance to avoid the delay of reading data from disks. This paper presents a brand-new method to create copies considering the selection of replicas set, the number of copies for each replica and the placement strategy of all copies. First, the popularities of all data are computed considering both the historical access records and the timeliness of the records. Then, replica set can be selected based on their recent popularities. Also, an enhanced Q-value scheme is proposed to assign the number of copies for each replica. Finally, a reasonable copies placement strategy is designed to meet the requirement of load balance. In addition, we present several experiments that compare the proposed method with techniques that use other replication management strategies. The results show that the proposed model has better performance than other algorithms in all respects. Moreover, the experiments based on different parameters also demonstrated the effectiveness and adaptability of the proposed algorithm.

Research on Decision-making Behavior Evolution of Government, Coal Mine Enterprises and Employees under Safe Education

The penalty type education mode for safety production is lack of systematic and comprehensive education thought, education system and education measures, which leads to frequent safety accidents. To promote the safety education level of China’s coal mining enterprises, in this study, the government, coal mine enterprises and employees are included in the evolutionary game model to explore the game relation and evolutionary path of the decision-making behavior among the three stakeholders in the safety management and education. It is proved by decision-making dynamic replication analysis, strategy evolution stability analysis and numerical simulated experiment verification that: the decision-making behavior of the government, coal mine enterprises and employees is the result of the mutual game playing among the three stakeholders. It is impossible for the government to leave alone the enterprises adopting the penalty type education mode for safety production. The incentive type education mode for safety production takes active education thought, education system, education measures and so on. It has become an effective driving factor to promote safety education level and motivate employees’ safety initiative. When certain conditions are met, the decision-making behavior of three stakeholders of the government, coal mine enterprises and employees can evolve into an ideal state. The evolution process of decision-making behavior of three stakeholders to the ideal state is influenced by many factors. The range of different factors has a differential effect on the convergence rate of the three stakeholders to the ideal state.

Dynamic Data Replication Based on Tasks scheduling for Cloud Computing Environment

Cloud computing provides IT resources (e.g., CPU, memory, network, storage, etc.) based on virtualization concepts and a pay-as-you-go principle. It comprises an accumulation of inter-related plus virtualized calculating resources which are managed by one or more amalgamated calculating resources. With the development of a computerized scientific workflow, the amount of data is increasing exponentially. Workflow scheduling and data replication have been considered the major challenges in cloud computing. Nevertheless, many researchers focus on scheduling or data replication separately. In this article, a combination of workflow scheduling based on the clustering of data and dynamic data replication strategies, has been introduced together and evaluates several performance metrics using a Cloudsim simulator. The aim of this proposed algorithm is to minimize the completion time and transfer time. The performance of this proposed algorithm has been evaluated using the CloudSim toolkit.

An enhanced bandwidth hierarchy based replication strategy for dynamic replication in data grid

An enhanced bandwidth hierarchy based replication strategy for dynamic replication in data grid

Keeping up with storage: Decentralized, write-enabled dynamic geo-replication

Large-scale applications are ever-increasingly geo-distributed. Maintaining the highest possible data locality is crucial to ensure high performance of such applications. Dynamic replication addresses this problem by dynamically creating replicas of frequently accessed data close to the clients. This data is often stored in decentralized storage systems such as Dynamo or Voldemort, which offer support for mutable data. However, existing approaches to dynamic replication for such mutable data remain centralized, thus incompatible with these systems. In this paper we introduce a write-enabled dynamic replication scheme that leverages the decentralized architecture of such storage systems. We propose an algorithm enabling clients to locate tentatively the closest data replica without prior request to any metadata node. Large-scale experiments on various workloads show a read latency decrease of up to 42% compared to other state-of-the-art, caching-based solutions.

A dynamic file replication based on CPU load and consistency mechanism in a trusted distributed environment

A dynamic file replication based on CPU load and consistency mechanism in a trusted distributed environment

Highly Available SDN Control of Flexi-Grid Networks With Network Function Virtualization-Enabled Replication

New trends and emerging requirements have driven the development of extensions to the path computation element (PCE) architecture beyond the computation of a set of constrained routes and associated resources between endpoints, given a network topology. Such extensions involve the use of a PCE for the control of network services, in which deploying a PCE as a centralized network controller facilitates the adoption of software-defined networking (SDN) principles while allowing a progressive migration of already existing deployments. A key requirement for the adoption of centralized control solutions is the ability to deploy a resilient, secure, dynamically configurable, adaptive, and highly available (virtualized) infrastructure supporting end-to-end services, including critical and vertical ones. Part of this infrastructure is the control plane functional elements (e.g., controllers), and the use of network function virtualization (NFV) is a enabler for the high availability of such elements while additionally reducing OPEX and CAPEX. NFV provides a feature-complete framework for the replication of software components that is a straightforward and commonly adopted approach to address the aforementioned requirement, but it implies the need for timely synchronization of databases between replicas. In this paper we present, implement, and validate an architecture for PCE and SDN control high availability, combining the virtualization of the control function by means of dynamic replication and the timely synchronization of their internal state using the PCEP and BGP-LS protocols. We experimentally validate the approach with a testbed, including a GMPLS/PCE control plane, and a replica management system implemented following the ETSI NFV framework, using the OpenStack cloud management software.

Fragmentation and Dynamic Replication Model in Multicloud by Data Hosting with Secured Data Sharing

Cloud Computing that can be provisioned for data hosting and data management. The cloud providers are hosting their data through different strategies. Enterprises and organizations are looking for service availability and to enhance the data reliability. Several Organizations has used single cloud-providers, but it has become less popular along with the customers due to risks of service availability failure. In this paper, division and dynamic replication of data management in the cloud storage has been proposed with few security aspects in which secured data sharing can be implemented. In replication model its divides a file into fragments and that file is replicated over the cloud nodes. In T-Coloring each node stores only single fragment even in case of successful attack, no meaningful information is revealed to the attacker and the nodes are placed with certain design. It reduces the access latency and improves data availability and data access. The system improves the data retrieval time and reconstructs the original cover. Cloud computing and storage solutions facilitate the users to store and process the data in third-party data centers. The privacy and security of group sharing data can be improved in this framework which combines proxy re-encryption, one time password verification and regular elimination of unwanted files. By applying proxy re-encryption, most computationally rigorous operations can be delegated to cloud servers without revealing any private information. One time secret key is one of the simplest and prominent type compared to the text based passwords, it can be utilized for securing access to accounts. The performance and results illustrates that our proposed scheme has the prospective to be successfully used for secure data sharing in the cloud

A HIGH AVAILABILITY CLUSTER-BASED REPLICA CONTROL PROTOCOL IN DATA GRID

Data replication is widely used to provide high data availability, and increase the performance of the distributed systems. Many replica control protocols have been proposed in distributed and grid environments that achieved both high performance and availability. However, the previously proposed protocols still require a bigger number of replicas for read and write operations which are not suitable for a large scale system such as data grid. In this paper, a new replica control protocol called Clusteringbased Hybrid (CBH) has been proposed for managing the data in grid environments. We analyzed the communication cost and data availability for the operations and compared CBH protocol with recently proposed replica control protocols called Dynamic Hybrid (DH) protocol and Diagonal Replication in 2D Mesh (DR2M) protocol. To evaluate CBH protocol, a simulation model was implemented using Java. Our results show that for the read operations, CBH protocol improves the performance of communication cost and data availability compared to the DH and DR2M protocols.

Evolutionary Game Analysis of Financial Regulation and Innovation Under Asymmetric Conditions

The loose financial supervision and excessive financial innovation will lead the country to heavy losses in financial crisis. So how to balance the relationship between financial supervision and financial innovation and make financial supervision and financial innovation to achieve complementary has become a hot issue. The financial supervision and financial institutions are a process of repeated game, but the research on them is still in a superficial stage. Therefore, this paper uses evolutionary game theory and method to analyze the dynamic replication system of asymmetric evolutionary game of two groups of financial institutions and regulators in China. And the evolutionary stable state of the system under different conditions is analyzed. The conclusion is that under different parameter values, the other side adopts different strategies, and the system will tend to different equilibrium states.

A prediction-based dynamic replication strategy for data-intensive applications

Data-intensive applications produce huge amount of data sets which need to be analyzed among geographically distributed nodes in grid computing environment. Data replication is essential in this environment to reduce the data access latency and to improve the data availability across several grid sites. In this work, an Intelligent Replica Manager (IRM) is designed and incorporated in the middleware of the grid for scheduling data-intensive applications. IRM uses a Multi-criteria based replication algorithm which considers multiple parameters like storage capacity, bandwidth and communication cost of the neighboring sites before taking decisions for the selection and placement of replica. Additionally, future needs of the grid site are predicted in advance using modified apriori algorithm, which is an association rule based mining technique. This IRM based strategy reduces the data availability time, data access time and make span. The simulation results prove that the proposed strategy outperforms the existing strategies.

Dynamic replication to reduce access latency based on fuzzy logic system

In a distributed environment, limited available bandwidth resources lead a high data access latency. Replication is a popular method that can upgrade the access performance and increase the data availability. However, unreasonable replication would cause over-consumption of system resources and finally a further deterioration on data access latency. So, In this paper, a theoretical model of access latency optimization with replication is presented firstly, which complement the blank space, and then a well-designed dynamic replication strategy is proposed, which consists of three algorithms: replica selection algorithm, replica layout algorithm and replica replacement algorithm. Replica selection algorithm selects the optimal replica with a hierarchical time cost based on the derivation of the theoretical model. Replica layout algorithm selects the optimal node for placing the replica based on the spatio-temporal locality of data access. Replica replacement algorithm, in which the fuzzy logic system is introduced originally, deletes replica when the available storage space is insufficient. FLSDR is tested by OptorSim and experimental results show that FLSDR achieves better performance in comparison with other algorithms in terms of mean job execution time, computing resource usage, number of data scheduling between clusters and number of replicas.

E2FS: an elastic storage system for cloud computing

In cloud storage, replication technologies are essential to fault tolerance and high availability of data. While achieving the goal of high availability, replication brings extra number of active servers to the storage system. Extra active servers mean extra power consumption and capital expenditure. Furthermore, the lack of classification of data makes replication scheme fixed at the very beginning. This paper proposes an elastic and efficient file storage called E2FS for big data applications. E2FS can dynamically scale in/out the storage system based on real-time demands of big data applications. We adopt a novel replication scheme based on data blocks, which provides a fine-grained maintenance of the data in the storage system. E2FS analyzes features of data and makes dynamic replication decision to balance the cost and performance of cloud storage. To evaluate the performance of proposed work, we implement a prototype of E2FS and compare it with HDFS. Our experiments show E2FS can outperform HDFS in elasticity while achieving guaranteed performance for big data applications.