Data Replication Mechanism Research Articles

Fault Tolerant Hash Join for Distributed Systems

Nowadays, enterprises are inclined to deploy data processing and analytical applications from well-equipped mainframes with highly available hardware components to commodity computers. Commodity machines are less reliable than expensive mainframes. Applications deployed on commodity clusters have to deal with failures that occur frequently. Mostly, these applications perform complex client queries with aggregation and join operations. The longer a query executes, the more it suffers from failures. It causes the entire work has to be re-executed. This paper presents a fault tolerant hash join (FTHJ) algorithm for distributed systems implemented in Apache Ignite. The FTHJ achieves fault tolerance by using a data replication mechanism, materializing intermediate computations. To evaluate FTHJ, we implemented the baseline, unreliable hash join algorithm. Experimental results show that FTHJ takes at least 30% less time to recover and complete join operation when a failure occurs during the execution. This paper describes how we reached a compromise between executing recovery tasks for the least amount of time and using additional resources.

MIX-RS: A Multi-Indexing System Based on HDFS for Remote Sensing Data Storage

A large volume of remote sensing (RS) data has been generated with the deployment of satellite technologies. The data facilitates research in ecological monitoring, land management and desertification, etc. The characteristics of RS data (e.g., enormous volume, large single-file size and demanding requirement of fault tolerance) make the Hadoop Distributed File System (HDFS) an ideal choice for RS data storage as it is efficient, scalable and equipped with a data replication mechanism for failure resilience. To use RS data, one of the most important techniques is geospatial indexing. However, the large data volume makes it time-consuming to efficiently construct and leverage. Considering that most modern geospatial data centres are equipped with HDFS-based big data processing infrastructures, deploying multiple geospatial indices becomes natural to optimise the efficacy. Moreover, because of the reliability introduced by high-quality hardware and the infrequently modified property of the RS data, the use of multi-indexing will not cause large overhead. Therefore, we design a framework called Multi-IndeXing-RS (MIX-RS) that unifies the multi-indexing mechanism on top of the HDFS with data replication enabled for both fault tolerance and geospatial indexing efficiency. Given the fault tolerance provided by the HDFS, RS data is structurally stored inside for faster geospatial indexing. Additionally, multi-indexing enhances efficiency. The proposed technique naturally sits on top of the HDFS to form a holistic framework without incurring severe overhead or sophisticated system implementation efforts. The MIX-RS framework is implemented and evaluated using real remote sensing data provided by the Chinese Academy of Sciences, demonstrating excellent geospatial indexing performance.

A Two-Level Cache Aware Adaptive Data Replication Mechanism for Shared LLC

A Two-Level Cache Aware Adaptive Data Replication Mechanism for Shared LLC

Cost-effective data replication mechanism modelling for cloud storage

Replication increases data availability and reduces access latency in cloud storage. Solutions prioritise system performance over replica production and storage costs. Under a pay-as-you-go model, cloud users incur rising fees for replica storage and consistency management. Before expanding node copies, we outline a dynamic replication technique. The suggested technique concentrates resources for effective distribution across all nodes. The cost-effectiveness of the proposed model is determined using threshold values; the proposed system reduced cloud replicas, reducing costs. Azure's $6 static solution has three nodes. The adaptive data replication management system reduces copies from three to two, saving memory and reducing price from $6 to $4. MATLAB and other programming languages are used. Azure is compared to Date Replication and Reproducer (DRR). The proposed solution is cheaper than Azure with three replicates.

Cost-effective data replication mechanism modelling for cloud storage

Cost-effective data replication mechanism modelling for cloud storage

A hardware-efficient dual-source data replication and local broadcast mechanism in distributed shared caches

Massive data access and edge computing have overwhelmed traditional edge computing devices. Researchers begin to study the deployment of many-core architectures with distributed shared caches in the chip of edge computing devices. The performance of traditional coherence protocols used in the distributed shared cache has become a potential bottleneck of system performance. To meet the requirements of low hardware overhead and high performance, we proposed a hardware-efficient dual-source data replication and local broadcast mechanism(DRLB). DRLB alleviates the inherent limitations of previous data replication mechanisms and reduces the overhead of coherence protocols. We verified the correctness and feasibility of DRLB by hardware implementation and proposed a hardware optimization of the non-blocking data replication mechanism. We evaluated DRLB and found that it can reduce the execution time by 16%, but only causes 0.82% of extra meta storage overhead, which also outperforms previous state-of-the-art data replication mechanism. What is more, the optimized DRLB can reduce the execution time by an average of 23%.

Data replication techniques in the Internet of Things: a systematic literature review

PurposeIn data grids, replication has been regarded as a crucial optimization strategy. Computing tasks are performed on IoT gateways at the cloud edges to obtain a prompt response. So, investigating the data replication mechanisms in the IoT is necessary. Henceforth, a systematic survey of data replication strategies in IoT techniques is presented in this paper, and some suggestions are offered for the upcoming works. In two key classifications, various parameters dependent on the analysis of the prevalent approaches are considered. The pros and cons associated with chosen strategies have been explored, and the essential problems of them have been presented to boost the future of more effective data replication strategies. We have also discovered gaps in papers and provided solutions for them.Design/methodology/approachProgress in Information Technology (IT) growth has brought the Internet of Things (IoT) into life to take a vital role in our everyday lifestyles. Big IoT-generated data brings tremendous data processing challenges. One of the most challenging problems is data replication to improve fault-tolerance, reliability, and accessibility. In this way, if the primary data source fails, a replica can be swapped in immediately. There is a significant influence on the IoT created by data replication techniques, but no extensive and systematic research exists in this area. There is still no systematic and full way to address the relevant methods and evaluate them. Hence, in the present investigation, a literature review is indicated on the IoT-based data replication from papers published until 2021. Based on the given guidelines, chosen papers are reviewed. After establishing exclusion and inclusion criteria, an independent systematic search in Google Scholar, ACM, Scopus, Eric, Science Direct, Springer link, Emerald, Global ProQuest, and IEEE for relevant studies has been performed, and 21(6 paper analyzed in section 1 and 15 paper analyzed in section 3) papers have been analyzed.FindingsThe results showed that data replication mechanisms in the IoT algorithms outperform other algorithms regarding impressive network utilization, job implementation time, hit ratio, total replication number, and the portion of utilized storage in percentage. Although a few ideas have been suggested that fix different facets of IoT data management, we predict that there is still space for development and more study. Thus, in order to design innovative and more effective methods for future IoT-based structures, we explored open research directions in the domain of efficient data processing.Research limitations/implicationsThe present investigation encountered some drawbacks. First of all, only certain papers published in English were included. It is evident that some papers exist on data replication processes in the IoT written in other languages, but they were not included in our research. Next, the current report has only analyzed the mined based on data replication processes and IoT keyword discovery. The methods for data replication in the IoT would not be printed with keywords specified. In this review, the papers presented in national conferences and journals are neglected. In order to achieve the highest ability, this analysis contains papers from major global academic journals.Practical implicationsTo appreciate the significance and accuracy of the data often produced by different entities, the article illustrates that data provenance is essential. The results contribute to providing strong suggestions for future IoT studies. To be able to view the data, administrators have to modify novel abilities. The current analysis will deal with the speed of publications and suggest the findings of research and experience as a future path for IoT data replication decision-makers.Social implicationsIn general, the rise in the knowledge degree of scientists, academics, and managers will enhance administrators' positive and consciously behavioral actions in handling IoT environments. We anticipate that the consequences of the present report could lead investigators to produce more efficient data replication methods in IoT regarding the data type and data volume.Originality/valueThis report provides a detailed literature review on data replication strategies relying on IoT. The lack of such papers increases the importance of this paper. Utilizing the responses to the study queries, data replication's primary purpose, current problems, study concepts, and processes in IoT are summarized exclusively. This approach will allow investigators to establish a more reliable IoT technique for data replication in the future. To the best of our understanding, our research is the first to provide a thorough overview and evaluation of the current solutions by categorizing them into static/dynamic replication and distributed replication subcategories. By outlining possible future study paths, we conclude the article.

Modified Data Storage and Replication Mechanism with Frequent Use-Case Based Indexing

The real time applications demands high speed and reliable data access from the remote database. An effective logical data management strategy that handles simultaneous connections with better performance negotiation is inevitable. This work considers an e-health care application that proposes MongoDB based modified indexing and performance tuning methods. To cope with certain high frequency use case and its performance mandates, a flexible and efficient logical data management may be preferred. By analysing the data dependency, data decomposition concerns and the performance requirements of the specific use case of the medical application, a logical schema may be customized on an ala-carte basis. This work focused on the flexible logical data modeling schemes and its performance factors of the NoSql DB. The efficiency of unstructured data base management in storing and retrieving the e-health care data was analysed with a web based tool. To enable faster data retrieval and query processing over the distributed nodes, a Spark based storage engine was built on top of the MongoDB based data storage management. With Spark tool, the database has been made distributed as master–slave structures with suitable data replication mechanisms. In such distributed database the fail-over also implemented with the suitable replication mechanism. This work considered MongoDB based flexible schema modeling and Spark based distributed computation with multiple chunks of data. The flexible data modeling scheme with MongoDB with the on-demand Spark based computation framework was proposed. To facilitate the eventual consistency, scalability aspects of the e-health care applications, use case based indexing was proposed. With the effective data management, faster query processing the horizontal scalability has been increased. The overall efficiency and scalability of the proposed logical data management approach was analysed. Through the simulation studies, the proposed approach has been claimed to boost the performance of the bigdata based application to a considerable extent.

BigDataSDNSim: A simulator for analyzing big data applications in software‐defined cloud data centers

AbstractThe integration and crosscoordination of big data processing and software‐defined networking (SDN) are vital for improving the performance of big data applications. Various approaches for combining big data and SDN have been investigated by both industry and academia. However, empirical evaluations of solutions that combine big data processing and SDN are extremely costly and complicated. To address the problem of effective evaluation of solutions that combine big data processing with SDN, we present a new, self‐contained simulation tool named BigDataSDNSim that enables the modeling and simulation of the big data management system YARN, its related programming models MapReduce, and SDN‐enabled networks in a cloud computing environment. BigDataSDNSim supports cost‐effective and easy to conduct experimentation in a controllable, repeatable, and configurable manner. The article illustrates the simulation accuracy and correctness of BigDataSDNSim by comparing the behavior and results of a real environment that combines big data processing and SDN with an equivalent simulated environment. Finally, the article presents two uses cases of BigDataSDNSim, which exhibit its practicality and features, illustrate the impact of data replication mechanisms of MapReduce in Hadoop YARN, and show the superiority of SDN over traditional networks to improve the performance of MapReduce applications.

Data replication mechanisms in the peer‐to‐peer networks

SummaryPeer‐to‐peer (P2P) network is a distributed network in which nodes with similar capabilities exchange information with each other. Due to the nature of the P2P network distribution, numerous network message transformation is required to exchange the data between nodes over the network, which may increase access latency. Because of the high amount of the stored data in the P2P networks, the replication of data is very important. A large amount of data is handled using data replication to increase data access, reduce access latency, and increase data availability. This issue has an important role in the P2P networks, but there is no complete and systematic research in this field. Therefore, this paper aims to provide a comprehensive study of data replication mechanisms in the P2P networks. The 1027 papers have been identified and have been reduced to 213 main studies using the paper selection process. Moreover, in this paper, the major advances are reviewed in four main groups (structured, unstructured, super‐peer, and hybrid networks), and the new challenges are also highlighted. Moreover, the open issues and guidelines for future studies are presented.

A Review of Static and Dynamic Data Replication Mechanisms for Distributed Systems

A Review of Static and Dynamic Data Replication Mechanisms for Distributed Systems

Selective Data Replication for Online Social Networks with Distributed Datacenters

Though the new OSN model, which deploys datacenters globally, helps reduce service latency, it causes higher inter-datacenter communication load. In Facebook, each datacenter has a full copy of all data, and the master datacenter updates all other datacenters, generating tremendous load in this new model. Distributed data storage, which only stores a user's data to his/her geographically closest datacenters mitigates the problem. However, frequent interactions between distant users lead to frequent inter-datacenter communication and hence long service latencies. In this paper, we aim to reduce inter-datacenter communications while still achieving low service latency. We first verify the benefits of the new model and present OSN typical properties that underlie the basis of our design. We then propose Selective Data replication mechanism in Distributed Datacenters ( $SD^3$ ). Since replicas need inter-datacenter data updates, datacenters in $SD^3$ jointly consider update rates and visit rates to select user data for replication; furthermore, $SD^3$ atomizes users’ different types of data (e.g., status update, friend post, music) for replication, ensuring that a replica always reduces inter-datacenter communication. $SD^3$ also incorporates three strategies to further enhance its performance: locality-aware multicast update tree, replica deactivation, and datacenter congestion control. The results of trace-driven experiments on the real-world PlanetLab testbed demonstrate the higher efficiency and effectiveness of $SD^3$ in comparison to other replication methods and the effectiveness of its three schemes.

In-Scratchpad Memory Replication

Scratchpad memories (SPMs) are widely employed in multicore embedded processors. Reliability is one of the major constraints in the embedded processor design, which is threatened with the increasing susceptibility of memory cells to multiple-bit upsets (MBUs) due to continuous technology down-scaling. This article proposes a low-cost and efficient data replication mechanism, called In-Scratchpad Memory Replication (ISMR), to correct MBUs in SPMs of multicore embedded processors. The main feature of ISMR is a smart controller, called Replication Management Unit (RMU), which is responsible for dynamically analyzing the activity of the SPM blocks at runtime and efficiently replicating the vulnerable SPM blocks into currently inactive SPM blocks. RMU exploits a 2-bit tag for each SPM block, where the value of each tag is determined by RMU according to the SPM access pattern. Accordingly, the proposed mechanism guarantees the replication of all vulnerable SPM blocks to provide error correction without decreasing the SPM utilization. To detect errors in SPM blocks, ISMR uses a 2-bit interleaved-parity code. As compared with the previous E-RAID 1 mechanism, the simulation results illustrate that for an 8-core embedded processor, the ISMR mechanism experiences 81% less energy consumption overhead and 48% less performance overhead.

DGridSim: a multi-model discrete-event simulator for real-time data grid systems

Data grid systems are utilized to share, manage, and process large data sets. On the other hand, an increasing number of applications with real-time constraints arise in several disciplines of science and engineering. The performance of a data grid system for real-time applications is highly dependent on the underlying job scheduling, data scheduling, and data replication algorithms and advance reservation mechanism. Thus, in the literature, there are numerous studies that propose solutions to the job/data scheduling, data replication, and advance reservation problems. In these studies, a number of simulators, emulators, or test beds have been used to evaluate the proposed algorithms. Furthermore, these simulators/emulators usually adopt fixed-grid models, which in turn dictate specific job/data scheduling and data replication mechanisms. In the literature, there is no unified framework for modeling grid systems with different architectures, which can allow researchers to develop new grid system models and evaluate them in a flexible manner. This paper presents a unique framework for modeling real-time data grid systems that attempts to unify a large class of job scheduling, data scheduling, and data replication algorithms based on several system services. Then, in order to enable the development of these algorithms under different system models, DGridSim is realized to be a multi-model discrete-event simulator, and its capabilities are exemplified by means of a set of simulation results. The main contribution of the research is DGridSim, which can model and simulate a variety of different data grid system models by means of several system services and their interactions.

Data dissemination scheme for distributed storage for IoT observation systems at large scale

In the emerging field of the Internet of Things (IoT), Wireless Sensor Networks (WSNs) have a key role to play in sensing and collecting measures on the surrounding environment. In the deployment of large scale observation systems in remote areas, when there is not a permanent connection with the Internet, WSNs are calling for replication and distributed storage techniques that increase the amount of data stored within the WSN and reduce the probability of data loss. Unlike conventional network data storage, WSN-based distributed storage is constrained by the limited resources of the sensors. In this paper, we propose a low-complexity distributed data replication mechanism to increase the resilience of WSN-based distributed storage at large scale. In particular, we propose a simple, yet accurate, analytical modeling framework and an extensive simulation campaign, which complement experimental results on the SensLab testbed. The impact of several key parameters on the system performance is investigated.

A Multi-Agent Based Data Replication Mechanism for Mobile Grid

Problem statement: In Mobile grid, the mobile devices can be effectively incorporated in to the Grid. It enables both the mobility of the users requesting access to a fixed grid and the resources that are themselves as part of the grid. There are number challenging issues for mobile grid; such as connectivity, availability, maintainability, consistency and fault tolerance. This study addresses the issue of data replication solution that maintains consistency, which improves the availability of replicated data in a small scale mobile grid environment. Approach: In this study we propose a structural design that improves the availability in a mobile grid environment by placing the replica in an optimized manner so that the performance of the mobile grid environment can be improved and a multi agent based approach to maintain the consistency between the replicated data effectively by propagating the latest updates intelligently. Results: The study presents the results depicting the advantageous of using agents in data replication, which includes reduction in data communication cost under different circumstances like change in mobility of nodes, read write ratio of nodes and replication schema. Conclusion: The proposed method will select the optimum number of locations to place the replica such that the maintenance overhead such as update propagation, consistency maintenance, storage cost and communication cost will be reduced.

A novel data replication mechanism in P2P VoD system

With the development of the Internet, high-quality streaming services, including Video-on-Demand, are more popular than ever with the help of P2P technologies. But peer-to-peer (P2P) on-demand streaming systems inevitably suffer from peer churn that is the inherent dynamic characteristic of overlay networks. With frequent peer departure and VCR operations, a large amount of media data cached on peer disks turn off-line and unavailable, which becomes the major reason of heavy server load. And the phenomenon has been proved by the system logs of self-developed P2P based Video-on-Demand platform, called GridCast. To address the above issues, a new proactive data replication mechanism is proposed and implemented into GirdCast. Based on the new mechanism, a peer can proactively replicate data chunks to stable cache servers for future sharing, when it has high possibility to leave the overlay. Two key heuristic algorithms are designed for departure prediction and replicating chunks selection. And the cache servers managements are also described in the submission. Trace driven simulations show that the mechanisms greatly decrease bandwidth load of media source server and improve the availability of chunks highly demanded but poorly provisioned by overlay peers.

Exploiting Virtual Coordinates for Improved Routing Performance in Sensor Networks

We present the Virtual Cord Protocol (VCP), which exploits virtual coordinates to provide efficient and failure tolerant routing and data management in sensor networks. VCP maintains a virtual cord interconnecting all the nodes in the network and which, operating similar to a Distributed Hash Table (DHT), provides means for inserting data fragments into sensor nodes and retrieving them. Furthermore, it supports service discovery using indirections. VCP uses two mechanisms for finding paths to nodes and associated data items: First, it relies on the virtual cord that always provides a path toward the destination. Second, locally available neighborhood information is exploited for greedy routing. Our simulation results show that VCP is able to find paths close to the shortest path (achieving a stretch ratio of less than 125 percent) with very low overhead. We also extended VCP with data replication mechanisms to improve failure handling. The routing performance of VCP, which clearly outperforms other ad hoc routing protocols such as Dynamic MANET On Demand (DYMO), is similar to other virtual addressing schemes, e.g., Virtual Ring Routing (VRR). However, we improved VCP to handle frequent node failures in an optimized way. The presented results outline the capabilities of VCP to handle such cases more efficiently compared to other protocols. We also compared the capabilities to reliably store and retrieve data in the network to Geographic Hash Tables (GHTs). VCP, in the worst case, performs similar to GHTs, but outperforms this protocol in most cases, especially when complex routing is involved.

The Impact of Data Replication on Job Scheduling Performance in Hierarchical Data Grid

In data-intensive applications data transfer is a primary cause of job execution delay. Data access time depends on bandwidth. The major bottleneck to supporting fast data access in Grids is the high latencies of Wide Area Networks and Internet. Effective scheduling can reduce the amount of data transferred across the internet by dispatching a job to where the needed data are present. Another solution is to use a data replication mechanism. Objective of dynamic replica strategies is reducing file access time which leads to reducing job runtime. In this paper we develop a job scheduling policy and a dynamic data replication strategy, called HRS (Hierarchical Replication Strategy), to improve the data access efficiencies. We study our approach and evaluate it through simulation. The results show that our algorithm has improved 12% over the current strategies.

SurePath: An Approach to Resilient Anonymous Routing

Current anonymous systems either use a small, fixed core set of mixes or randomly choose a sequence of peer nodes to form anonymous paths/tunnels. The resulting paths are fragile and shortlived -- that is, a path fails if one of the mixes/nodes fails or leaves the system. In this paper, we propose SurePath, a failure-resilient tunneling approach for anonymity built on distributed hash tables (DHTs). SurePath aims to make anonymous tunnels fault-tolerant to node failures. The basic idea behind SurePath is to decouple anonymous tunnels from ”fixed” nodes and form anonymous tunnels from dynamic mix nodes by relying on DHT routing infrastructure and data replication mechanism. We have implemented SurePath in Java on FreePastry 1.3. We also provide analysis of anonymity and performance evaluation for SurePath.