Distributed Data Storage System Research Articles

Assurance of Cyber Resistance of the Distributed Data Storage Systems Using the Blockchain Technology

The article studies the architectures of the modern systems of decentralized data storage and processing, applicability of the blockchain technology in these systems, existing security threats in comparison with the decentralized systems, and methods of security assurance allowing avoidance of these threats.

Towards Distributed Data Management in Fog Computing

In the emerging area of the Internet of Things (IoT), the exponential growth of the number of smart devices leads to a growing need for efficient data storage mechanisms. Cloud Computing was an efficient solution so far to store and manipulate such huge amount of data. However, in the next years it is expected that Cloud Computing will be unable to handle the huge amount of the IoT devices efficiently due to bandwidth limitations. An arising technology which promises to overwhelm many drawbacks in large-scale networks in IoT is Fog Computing. Fog Computing provides high-quality Cloud services in the physical proximity of mobile users. Computational power and storage capacity could be offered from the Fog, with low latency and high bandwidth. This survey discusses the main features of Fog Computing, introduces representative simulators and tools, highlights the benefits of Fog Computing in line with the applications of large-scale IoT networks, and identifies various aspects of issues we may encounter when designing and implementing social IoT systems in the context of the Fog Computing paradigm. The rationale behind this work lies in the data storage discussion which is performed by taking into account the importance of storage capabilities in modern Fog Computing systems. In addition, we provide a comprehensive comparison among previously developed distributed data storage systems which consist of a promising solution for data storage allocation in Fog Computing.

Privacy and security of big data in cyber physical systems using Weibull distribution-based intrusion detection

The volume of data collected from cyber physical systems (CPS) is huge, and we use big data techniques to manage and store the data. Big data in CPS is concerned with the massive heterogeneous data streams, which are acquired from autonomous sources and computed in distributed data storage system. In order to handle the size, complexity and rate of availability of data, it requires new techniques that can inspect and interpret useful knowledge from large streams of information, which impose challenges on the design and management of CPS in multiple aspects such as performance, energy efficiency, security, privacy, reliability, sustainability, fault tolerance, scalability and flexibility. This paper focuses on the security and privacy aspects in managing big data for CPS and reviews recent challenges in data privacy. We also present a protection framework for intrusion detection and analyze the performance parameters, reliability and failure rate in a malicious big data context.

Activity of public control entities and development of distributed computing and distributed data storage systems

Introduction. This article is devoted to legal regulation of digital assets turnover, utilization possibilities of distributed computing and distributed data storage systems in activities of public authorities and entities of public control. The author notes that some national and foreign scientists who study a “blockchain” technology (distributed computing and distributed data storage systems) emphasize its usefulness in different activities. Data validation procedure of digital transactions, legal regulation of creation, issuance and turnover of digital assets need further attention.Materials and methods. The research is based on common scientific (analysis, analogy, comparing) and particular methods of cognition of legal phenomena and processes (a method of interpretation of legal rules, a technical legal method, a formal legal method and a formal logical one).Results of the study. The author conducted an analysis which resulted in finding some advantages of the use of the “blockchain” technology in the sphere of public control which are as follows: a particular validation system; data that once were entered in the system of distributed data storage cannot be erased or forged; absolute transparency of succession of actions while exercising governing powers; automatic repeat of recurring actions. The need of fivefold validation of exercising governing powers is substantiated. The author stresses that the fivefold validation shall ensure complex control over exercising of powers by the civil society, the entities of public control and the Russian Federation as a federal state holding sovereignty over its territory. The author has also conducted a brief analysis of judicial decisions concerning digital transactions.Discussion and conclusion. The use of the distributed data storage system makes it easier to exercise control due to the decrease of risks of forge, replacement or termination of data. The author suggests defining digital transaction not only as some actions with digital assets, but also as actions toward modification and addition of information about legal facts with a purpose of its establishment in the systems of distributed data storage. The author suggests using the systems of distributed data storage for independent validation of information about activities of the bodies of state authority. In the author’s opinion, application of the “blockchain” technology may result not only in the increase of efficiency of public control, but also in the creation of a new form of public control – automatic control. It is concluded there is no legislation basis for regulation of legal relations concerning distributed data storage today.

Byzantine fault-tolerant and semantic-driven consensus protocol

To provide fault tolerance, modern distributed storage systems use specialized network topologies and consensus protocols that create high overheads. The main disadvantage of existing specialized topologies is a difficulty to implement an efficient data placement that takes into account locality of the data. In scientific problems very often it is necessary to provide the semantic proximity of the data at the nodes of the distributed system. Core drawback of modern consensus protocols is that most of them use messaging through broadcasting, which is impossible for large amounts of data and storage nodes. With more stringent requirements for a fault tolerance, e.g. requirement for resistance to Byzantine errors [1], ensuring of data localization becomes nearly impossible. The purpose of this paper is to implement a distributed consensus protocol that preserves data localization and is resistant to Byzantine errors. Theoretical tasks: a) consider how to implement distributed services based on finite state machines; b) consider how to ensure Byzantine fault tolerance; c) consider methods for organizing the semantic proximity of data. Practical tasks: a) develop ways to jointly ensure the Byzantine fault tolerance and locality of data; b) study the shortcomings of the joint provision of Byzantine fault tolerance and locality of data; c) develop a protocol of Byzantine consensus using semantic links between data; d) analyze the performance of the developed protocol within the framework of applied tasks. The object of the study is a distributed data storage system with the requirement for Byzantine fault tolerance. The subject of this study is the provision of Byzantine fault tolerance in conjunction with the use of semantic links between the data. The subject of this study has never been the subject of a special scientific study. Separate scientific foundations of this problematic set in the present study were developed in separate works on Byzantine fault tolerance, local hash methods and dimensional reduction.

AR-RRNS: Configurable reliable distributed data storage systems for Internet of Things to ensure security

Benefits of Internet of Things and cloud–fog-edge computing are associated with the risks of confidentiality, integrity, and availability related with the loss of information, denial of access for a long time, information leakage, conspiracy and technical failures. In this article, we propose a configurable, reliable, and confidential distributed data storage scheme with the ability to process encrypted data and control results of computations. Our system utilizes Redundant Residue Number System (RRNS) with new method of error correction codes and secret sharing schemes. We introduce the concept of an approximate value of a rank of a number (AR), which allows us to reduce the computational complexity of the decoding from RNS to binary representation, and size of the coefficients. Based on the properties of the approximate value and arithmetic properties of RNS, we introduce AR-RRNS method for error detection, correction, and controlling computational results. We provide a theoretical basis to configure probability of information loss, data redundancy, speed of encoding and decoding to cope with different objective preferences, workloads, and storage properties. Theoretical analysis shows that by appropriate selection of RRNS parameters, the proposed scheme allows not only increasing safety, reliability, and reducing an overhead of data storage, but also processing of encrypted data.

REGENERATING CODETECHNIQUE IN DISTRIBUTED STORAGE

Abstract The Increasing need of storing large amounts of data presents a new challenge. One way to address this challenge is to use distributed data storage system. One of the strategies implemented in the distributed data storage system is using the technique of regenerating code. The code used in this technique is based on the algebraic structure of fields. Some studies have also been carried out to create code that is based on the other algebraic structure namely module. In this study, we attempted to assess the implementation of the code module at regenerating technique code. The study showed there is a potential properties code based on module that can be used in regenerating code technique. Keywords: Distributed storage, regenerating code technique, module code

On the Communication Cost of MDS Erasure Codes in Distributed Storage Systems

Distributed storage systems store some redundant data to keep the degree of availability of the stored data constant and also to increase the system's resistance against failures. This type of systems usually use pure replication or methods based on RAID systems as redundancy schemes. In this paper, we study the communication cost of a distributed data storage system using Maximum Distance Separable (MDS) erasure codes. Our focus is reduction of the cost of one-to-many communication used in data reconstruction/repair initialization and update operations. We propose the use of two different communication approaches on the area of distributed storage systems for the above operations; Steiner tree approach and multi-shortest path approach. We also analyse these two communication approaches empirically and theoretically. Our theoretical results indicate that Steiner tree approach has lower message usage, whereas, multi-shortest path approach has lower time usage for data reconstruction/repair initialization operations. On the other hand, Steiner tree approach has better message and time metrics for the data update process. Furthermore, our experimental results support these theoretical results. Thus, users can choose between the two approaches depending on their needs and priorities.

Technologies of distributed data storage systems for numerical simulation of materials

A method for structuring experimental data for applications in software packages for simulating materials is considered. A unified transport file (UTF) is proposed for data exchange between network nodes. In the study of data structuring algorithms, the data retrieval rate in the storage system was estimated under conditions of increasing data base content.

PRELIMINARY STUDY ON APPLICATION OF MAX PLUS ALGEBRA IN DISTRIBUTED STORAGE SYSTEM THROUGH NETWORK CODING

The increasing need in techniques of storing big data presents a new challenge. One way to address this challenge is the use of distributed storage systems. One strategy that implemented in distributed data storage systems is the use of Erasure Code which applied to network coding. The code used in this technique is based on the algebraic structure which is called as vector space. Some studies have also been carried out to create code that is based on other algebraic structures such as module. In this study, we are going to try to set up a code based on the algebraic structure which is a generalization of the module that is semimodule by utilizing the max operations and sum operations at max plus algebra. The results of this study indicate that the max operation and the addition operation on max plus algebra cannot be used to establish a semimodule code, but by modifying the operation "+" as "min", we get a code based on semimodule. Keywords: code, distributed storage systems, network coding, semimodule, max plus algebra

Distributed Data Storage Systems for Data Survivability in Wireless Sensor Networks using Decentralized Erasure Codes

Achieving reliability in Wireless Sensor Networks (WSNs) is challenging due to the limited resources available. In this study, we investigate the design of data survivability schemes using decentralized storage systems in WSNs. We propose a data storage system design based on Decentralized Erasure Codes (DEC) that features a simple and decentralized construction of the target code. The proposed framework allows sensor nodes to cooperate to build an erasure code-based storage that can tolerate a given failure/erasure rate. Code construction and decoding can both be performed randomly allowing for a distributed operation with no prior setup or coordination between source nodes. Further, we present two approaches that utilize Random Linear Network Coding (RLNC) to enhance the proposed scheme in order to achieve energy efficiency. We present the theoretical basis of the schemes then validate and evaluate their performance through simulations.

Reliability of Erasure Coded Storage Systems: A Combinatorial-Geometric Approach

We consider the probability of data loss, or equivalently, the reliability function for an erasure coded distributed data storage system under worst case conditions. Data loss in an erasure coded system depends on probability distributions for the disk repair duration and the disk failure duration. In previous works, the data loss probability of such systems has been studied under the assumption of exponentially distributed disk failure and disk repair durations, using well-known analytic methods from the theory of Markov processes. These methods lead to an estimate of the integral of the reliability function. Here, we address the problem of directly calculating the data loss probability for general repair and failure duration distributions. A closed limiting form is developed for the probability of data loss and it is shown that the probability of the event that a repair duration exceeds a failure duration is sufficient for characterizing the data loss probability. For the case of constant repair duration, we develop an expression for the conditional data loss probability given the number of failures experienced by a each node in a given time window. We do so by developing a geometric approach that relies on the computation of volumes of a family of polytopes that are related to the code. An exact calculation is provided and an upper bound on the data loss probability is obtained by posing the problem as a set avoidance problem. Theoretical calculations are compared to simulation results.

Low-cost high performance distributed data storage for multi-channel observations

The New Vacuum Solar Telescope (NVST) is a 1-m solar telescope that aims to observe the fine structures in both the photosphere and the chromosphere of the Sun. The observational data acquired simultaneously from one channel for the chromosphere and two channels for the photosphere bring great challenges to the data storage of NVST. The multi-channel instruments of NVST, including scientific cameras and multi-band spectrometers, generate at least 3 terabytes data per day and require high access performance while storing massive short-exposure images. It is worth studying and implementing a storage system for NVST which would balance the data availability, access performance and the cost of development. In this paper, we build a distributed data storage system (DDSS) for NVST and then deeply evaluate the availability of real-time data storage on a distributed computing environment. The experimental results show that two factors, i.e., the number of concurrent read/write and the file size, are critically important for improving the performance of data access on a distributed environment. Referring to these two factors, three strategies for storing FITS files are presented and implemented to ensure the access performance of the DDSS under conditions of multi-host write and read simultaneously. The real applications of the DDSS proves that the system is capable of meeting the requirements of NVST real-time high performance observational data storage. Our study on the DDSS is the first attempt for modern astronomical telescope systems to store real-time observational data on a low-cost distributed system. The research results and corresponding techniques of the DDSS provide a new option for designing real-time massive astronomical data storage system and will be a reference for future astronomical data storage.

Распределенные системы хранения данных: анализ, классификация и выбор

This paper proposes an analysis of various distributed data storage systems and possible solutions to basic problems of the subject area, in particular, the issue of system scaling, data consistency, availability and partition tolerance. Performed analysis has allowed to reveal consistent patterns with an attempt to classify systems based on various parameters, in particular, the presence or absence of specific functions and mechanisms. The choices of the distributed data storage systems are based on data analysis and classification.

Self-repairing codes

Networked distributed data storage systems are essential to deal with the needs of storing massive volumes of data. Dependability of such a system relies on its fault tolerance (data should be available in case of node failures) as well as its maintainability (its ability to repair lost data to ensure redundancy replenishment over time). Erasure codes provide a storage efficient alternative to replication based redundancy in storage systems, ensuring the same fault tolerance at a lower storage overhead cost. Traditional erasure codes however have the drawback of entailing high communication overhead for maintenance, when encoded fragments are lost due to storage device failures, and need to be replenished in new nodes. We propose a new family of erasure codes called self-repairing codes (SRC) taking into account the peculiarities of distributed storage systems, specifically to improve its maintainability by `localizing' the repairs. SRC have the property that encoded fragments can be repaired directly from other small subsets of (typically 2 or 3) encoded fragments. These code properties allow bandwidth efficient and fast recovery even in the presence of multiple failures, in turn translating into better system robustness. A concrete family of such locally repairable codes, namely, homomorphic SRC are proposed and various aspects and properties of the same are studied in detail and compared--quantitatively or qualitatively (as may be suitable) with respect to other codes including traditional erasure codes as well as some recent representative codes designed specifically for storage applications.

On construction of a distributed data storage system in cloud

In the past, people have focused on cluster computing and grid computing. Now, however, this focus has shifted to cloud computing. Irrespective of what techniques are used, there are always storage requirements. The challenge people face in this area is the huge amount of data to be stored, and its complexity. People are now using many cloud applications. As a result, service providers must serve increasingly more people, causing more and more connections involving substantially more data. These problems could have been solved in the past, but in the age of cloud computing, they have become more complex. This paper focuses on cloud computing infrastructure, and especially data services. The goal of this paper is to implement a high performance and load balancing, and able-to-be-replicated system that provides data storage for private cloud users through a virtualization system. This system extends and enhances the functionality of the Hadoop distributed system. The proposed approach also implements a resource monitor of machine status factors such as CPU, memory, and network usage to help optimize the virtualization system and data storage system. To prove and extend the usability of this design, a synchronize app was also developed running on Android based on our distributed data storage.

Optimal Locally Repairable Linear Codes

Linear erasure codes with local repairability are desirable for distributed data storage systems. An [n, k, d] code having all-symbol (r, \delta})-locality, denoted as (r, {\delta})a, is considered optimal if it also meets the minimum Hamming distance bound. The existing results on the existence and the construction of optimal (r, {\delta})a codes are limited to only the special case of {\delta} = 2, and to only two small regions within this special case, namely, m = 0 or m >= (v+{\delta}-1) > ({\delta}-1), where m = n mod (r+{\delta}-1) and v = k mod r. This paper investigates the existence conditions and presents deterministic constructive algorithms for optimal (r, {\delta})a codes with general r and {\delta}. First, a structure theorem is derived for general optimal (r, {\delta})a codes which helps illuminate some of their structure properties. Next, the entire problem space with arbitrary n, k, r and {\delta} is divided into eight different cases (regions) with regard to the specific relations of these parameters. For two cases, it is rigorously proved that no optimal (r, {\delta})a could exist. For four other cases the optimal (r, {\delta})a codes are shown to exist, deterministic constructions are proposed and the lower bound on the required field size for these algorithms to work is provided. Our new constructive algorithms not only cover more cases, but for the same cases where previous algorithms exist, the new constructions require a considerably smaller field, which translates to potentially lower computational complexity. Our findings substantially enriches the knowledge on (r, {\delta})a codes, leaving only two cases in which the existence of optimal codes are yet to be determined.

Coding Schemes for Distributed Storage Systems: Implementation and Improvements

Distributed data storage systems are used to store data reliably over a distributed collection of storage locations, called peers. Coding schemes are used to store a portion of the data in the peers ensuring the complete retrieval of data, during peer failures. This has applications in various areas like Wireless Networks, Sensor Networks etc. In this framework we consider a large file to be stored in a distributed manner over few peers of limited capacity. Each peer stores a portion of the coded data, without the knowledge of the contents of other peers. Random Coding is one of the coding schemes used for this. In [1] coding coefficients are chosen randomly from a finite field to encode the data. The encoding is basically a linear combination of file pieces (pieces are elements of finite fields). The data downloader downloads these coded data from several peers and decodes to get the original data. The decoding is basically solving a system of linear equations over a finite field, which is the most time consuming step in the whole process. We give a simple C++ implementation of the schemes in [1] and plot the results. We are trying to find a scheme where coding vectors can be chosen such that the decoding complexity is reduced significantly. Also in a dynamic setting where nodes enter and leave system intermittently, are discussed.

Highly available transactions

To minimize network latency and remain online during server failures and network partitions, many modern distributed data storage systems eschew transactional functionality, which provides strong semantic guarantees for groups of multiple operations over multiple data items. In this work, we consider the problem of providing Highly Available Transactions (HATs): transactional guarantees that do not suffer unavailability during system partitions or incur high network latency. We introduce a taxonomy of highly available systems and analyze existing ACID isolation and distributed data consistency guarantees to identify which can and cannot be achieved in HAT systems. This unifies the literature on weak transactional isolation, replica consistency, and highly available systems. We analytically and experimentally quantify the availability and performance benefits of HATs---often two to three orders of magnitude over wide-area networks---and discuss their necessary semantic compromises.

A Grid-enabled web platform for integrated digital biobanking in paediatrics

A Grid-enabled web platform for integrated digital biobanking in paediatrics