Erasure Coding Techniques Research Articles

An Investigation on Reed-Solomon Codes as Erasure Coding Technique on Its Properties and Utilizations

Erasure coding is an essential part of cloud computing, which is an important technology for effective data storage for recovering data that may be lost due to various reasons, there are various erasure coding techniques in the market. In this paper, linear MDS codes, which is a branch of erasure coding, will be investigated on their performance and usage. This paper will focus on the Reed-Solomon Code, which is the most implemented form of linear MDS codes, on three different aspects: 1) the methodologies of the encoding and decoding operations; 2) the pros and cons of different forms of Reed-Solomon Codes; 3) the different ways that different Reed-Solomon Codes are being employed. Moreover, the paper includes the definition of general Cloud Computing for the audience to understand its importance, how the erasure coding acts like a fault tolerance system of Cloud Computing, and how different kinds of Reed-Solomon code perform on tolerating erasures in cloud storage failures.

Cache-Based Matrix Technology for Efficient Write and Recovery in Erasure Coding Distributed File Systems

With the development of various information and communication technologies, the amount of big data has increased, and distributed file systems have emerged to store them stably. The replication technique divides the original data into blocks and writes them on multiple servers for redundancy and fault tolerance. However, there is a symmetrical space efficiency problem that arises from the need to store blocks larger than the original data. When storing data, the Erasure Coding (EC) technique generates parity blocks through encoding calculations and writes them separately on each server for fault tolerance and data recovery purposes. Even if a specific server fails, original data can still be recovered through decoding calculations using the parity blocks stored on the remaining servers. However, matrices generated during encoding and decoding are redundantly generated during data writing and recovery, which leads to unnecessary overhead in distributed file systems. This paper proposes a cache-based matrix technique that uploads the matrices generated during encoding and decoding to cache memory and reuses them, rather than generating new matrices each time encoding or decoding occurs. The design of the cache memory applies the Weighting Size and Cost Replacement Policy (WSCRP) algorithm to efficiently upload and reuse matrices to cache memory using parameters known as weights and costs. Furthermore, the cache memory table can be managed efficiently because the weight–cost model sorts and updates matrices using specific parameters, which reduces replacement cost. The experiment utilized the Hadoop Distributed File System (HDFS) as the distributed file system, and the EC volume was composed of Reed–Solomon code with parameters (6, 3). As a result of the experiment, it was possible to reduce the write, read, and recovery times associated with encoding and decoding. In particular, for up to three node failures, systems using WSCRP were able to reduce recovery time by about 30 s compared to regular HDFS systems.

Erasure-Coding-Based Storage and Recovery for Distributed Exascale Storage Systems

Various techniques have been used in distributed file systems for data availability and stability. Typically, a method for storing data in a replication technique-based distributed file system is used, but due to the problem of space efficiency, an erasure-coding (EC) technique has been utilized more recently. The EC technique improves the space efficiency problem more than the replication technique does. However, the EC technique has various performance degradation factors, such as encoding and decoding and input and output (I/O) degradation. Thus, this study proposes a buffering and combining technique in which various I/O requests that occurred during encoding in an EC-based distributed file system are combined into one and processed. In addition, it proposes four recovery measures (disk input/output load distribution, random block layout, multi-thread-based parallel recovery, and matrix recycle technique) to distribute the disk input/output loads generated during decoding.

Slicing, Tokenization, and Encryption Based Combinational Approach to Protect Data-at-Rest in Cloud Using TF-Sec Model

Data in the Cloud, which applies to data as a cloud service provider (CSP), transmits stores, or manages it. The company will enforce the same definition of data usage while the data is resident within the enterprise and thus extend the required cryptographic security criteria to data collected, exchanged, or handled by CSP. The CSP Service Level Agreements cannot override the cryptographic access measures. When the data is transferred securely to CSP, it can be securely collected, distributed, and interpreted. Data at the rest position applies to data as it is processed internally in organized and in the unstructured ways like databases and file cabinets. The Data at the Rest example includes the use of cryptography for preserving the integrity of valuable data when processed. For cloud services, computing takes multiple forms from recording units, repositories, and many unstructured items. This paper presents a secure model for Data at rest. The TF-Sec model suggested is planned for use with Slicing, Tokenization, and Encryption. The model encrypts the given cloud data using AES 256 encryption, and then the encrypted block is sliced into the chunks of data fragments using HD-Slicer. Then it applies tokenization algorithm TKNZ to each chunk of data, applies erasure coding technique to tokens, applies the data dispersion technique to scramble encrypted data fragments, and allocates to storage nodes of the multiple CSP. In taking the above steps, this study aims to resolve the cloud security problems found and to guarantee the confidentiality of their data to cloud users due to encryption of data fragments would be of little benefit to a CSP.

Minority Disk Failure Prediction Based on Transfer Learning in Large Data Centers of Heterogeneous Disk Systems

The storage system in large scale data centers is typically built upon thousands or even millions of disks, where disk failures constantly happen. A disk failure could lead to serious data loss and thus system unavailability or even catastrophic consequences if the lost data cannot be recovered. While replication and erasure coding techniques have been widely deployed to guarantee storage availability and reliability, disk failure prediction is gaining popularity as it has the potential to prevent disk failures from occurring in the first place. Recent trends have turned toward applying machine learning approaches based on disk SMART attributes for disk failure predictions. However, traditional machine learning (ML) approaches require a large set of training data in order to deliver good predictive performance. In large-scale storage systems, new disks enter gradually to augment the storage capacity or to replace failed disks, leading storage systems to consist of small amounts of new disks from different vendors and/or different models from the same vendor as time goes on. We refer to this relatively small amount of disks as minority disks. Due to the lack of sufficient training data, traditional ML approaches fail to deliver satisfactory predictive performance in evolving storage systems which consist of heterogeneous minority disks. To address this challenge and improve the predictive performance for minority disks in large data centers, we propose a minority disk failure prediction model named TLDFP based on a transfer learning approach. Our evaluation results in two realistic datasets have demonstrated that TLDFP can deliver much more precise results and lower additional maintenance cost, compared to four popular prediction models based on traditional ML algorithms and two state-of-the-art transfer learning methods.

Video Streaming Over Vehicular Ad Hoc Networks Using Erasure Coding

Video streaming over vehicular ad hoc networks (VANETs) is an attractive application that provides many useful services for drivers. However, many problems need resolving, most of which are related to packet delay and packet loss. Due to the specific environment of VANETs, it is difficult to apply conventional protocols originally developed for the Internet, such as Real-time Transport Protocol (RTP). Previous research works have offered many new protocols to solve this problem. However, most of them cannot make full use of existing Internet video-streaming resources like RTP players and servers. In this paper, we propose the converter model to solve this compatibility issue. Based on this model, we first modify the RTP using the erasure coding (EC) technique to address the high packet loss rate of VANETs. This protocol is known as EC-RTP. Then, we develop two converters; the first stands on the boundary between the Internet and VANETs. It receives the RTP packets from the Internet and translates them to EC-RTP packets, which are shared between vehicles. The second converter receives EC-RTP packets, translates them back to RTP packets, and then sends them to the RTP player.

Packet Delivery Ratio/Cost in MANETs With Erasure Coding and Packet Replication

This paper studies the performance of a general two-hop relay (2HR)- $(x,w,f) $ packet delivery scheme that combines both erasure coding and packet replication techniques in mobile ad hoc networks (MANETs). Under this packet delivery scheme, a group of $x$ packets is first encoded into $w(w\geq x) $ coded packets using erasure coding, and each coded packet is then replicated to at most $f$ distinct relay nodes that can help to forward the coded packets to its destination node. The original packets can be recovered when $x$ distinct coded packets reach the destination node. To understand the packet delivery process under the 2HR- $(x,w,f) $ scheme, we develop a multidimensional Markov chain framework, and based on this analytical expressions on the packet delivery ratio and corresponding expected packet delivery cost are further derived. Finally, extensive simulation and numerical studies are conducted to illustrate the efficiency of the developed theoretical models and to illustrate our findings. Our results indicate that the replication parameter $f$ should be carefully selected in order to obtain a high packet-delivery-ratio performance while maintaining a relatively low delivery cost.

Delay control in MANETs with erasure coding and f-cast relay

Packet delay control in mobile ad hoc networks (MANETs) is critical to support delay-sensitive applications in such networks. By combining erasure coding and packet redundancy techniques, this paper proposes a general two-hop relay algorithm 2HR- $$(x,\tau ,f)$$ ( x , ? , f ) for a flexible control of packet delivery delay in MANETs, where a group of x packets in source node are first encoded into $$x\cdot \tau$$ x · ? encoded packets based erasure coding, and each encoded packet is then delivered to at most f distinct relay nodes (f-cast) that will help to forward the encoded packet to destination node. To understand the delay performance in a 2HR- $$(x,\tau ,f)$$ ( x , ? , f ) MANET, we then develop a discrete time multi-dimensional Markov chain model to depict the packet delivery process in the network, based on which closed-form results on mean and variance of packet delivery delay are further derived. Finally, extensive simulation and theoretical results are provided to illustrate the efficiency of our delay models as well as the capability of the 2HR- $$(x,\tau ,f)$$ ( x , ? , f ) algorithm in delay control.

Improving Network Performances by Integrating Erasure Coding and Game Theory Techniques in Random Access Protocols

This research focuses on network performance and how to solve the problem of low throughput in the Aloha Medium Access Control (MAC) protocol and its derivatives. For this purpose, we propose two complementary solutions. The first consists of the integration of the erasure coding scheme in this protocol to recover collided packets and to reduce the rate of collision between transmitted packets. Here, since each node sends N coded packets instead of the k original packets, we have (<em>N-k</em>) redundant packets. The introduction of redundancy and subsequently structuring it in an exploitable manner, allows serious errors injected by the channel to be corrected. However, if each node attempts to achieve its best output without regard for the other nodes’ actions, we add a complementary solution, which is based on the game theory technique of acquiring network equilibrium. This makes the network stronger and able to resist many collisions.

Performance analysis of CCSDS File Delivery Protocol and erasure coding techniques in deep space environments

The rising demand for multimedia services even in hazardous environments, such as space missions and military theatres, and the consequent need of proper internetworking technologies have revealed the performance limits experienced by TCP protocol over long-delay and lossy links and highlighted the importance of the communication features provided by the protocol architectures proposed by the Consultative Committee for Space Data Systems (CCSDS). This paper proposes a CCSDS File Delivery Protocol (CFDP) extension, based on the implementation of erasure coding schemes, within the CFDP itself, in order to assure high reliability to the data communication even in presence of very critical conditions, such as hard shadowing, deep-fading periods and intermittent links. Different encoding techniques are considered and various channel conditions, in terms of Bit Error Ratio and bandwidth values, are tested.