Distributed Storage Networks Research Articles

Investigating Utilization of Activated Distributed Storage Networks for Peak Flow Reduction Using Stochastic Storm Transposition

Investigating Utilization of Activated Distributed Storage Networks for Peak Flow Reduction Using Stochastic Storm Transposition

Robust-DSN: A Hybrid Distributed Replication and Encoding Network Grouped with a Distributed Swarm Workflow Scheduler

In many distributed applications such as the Internet of Things (IoT), large amounts of data are being generated that require robust storage solutions. Traditional cloud solutions, although efficient, often lack trust and transparency because of centralized management. To address these issues, we present Robust-DSN, a distributed storage network leveraging the hybrid distributed replication and encoding network (HYDREN) and the distributed swarm workflow scheduler (DSWS) as its main components. Our system uses an interplanetary file system (IPFS) as an underlay storage network and segments it into multiple regions to distribute the failure domain and improve the data’s proximity to users. HYDREN incorporates Reed–Solomon encoding and distributed replication to improve file availability, while DSWS optimizes resource allocation across the network. The uploaded file is encoded into chunks and distributed across distinct optimal nodes leveraging lightweight multithreading. Additionally, Robust-DSN verifies the integrity of all chunks by preserving the hashes when uploading and validating each chunk while downloading. The proposed system provides a comprehensive solution for resilient distributed data storage, focusing on the key challenges of data availability, integrity, and performance. The results reveal that compared with a state-of-the-art system, the proposed system improves file recovery by 15%, even with a 50% peer failure rate. Furthermore, with replication factor 4 and the same failure resilience as IPFS, it saves 50% storage and enhances file recovery by 8%. Robust-DSN acts as a distributed storage platform for emerging technologies, expanding storage system capabilities in a wide range of distributed applications.

Secure IoT data management and sharing architecture for information security using cryptographic technique

The massive amounts of data produced and gathered by smart devices through the internet support a wide range of applications, considerably improving our daily lives. Data sharing among smart devices must be safeguarded due to the sensitivity of the data involved in the transmission. The Internet of Things (IoT) environment must be protected from unauthorised access due to a variety of variables, including its attractiveness to cybercriminals, previous successful cyber-attacks, and consumers’ perceptions of security and reliability. Blockchain technology appears to be one promising technology that appears to address these security challenges extremely effectively. However, given the volume and rate at which smart devices generate data, Blockchain appears to be inefficient for storing it. The pace of data collection in the IoT context and the speed of transaction confirmation in the Blockchain network are the two key elements behind this. We connect the Blockchain and the Inter-Planetary File System (IPFS) in this study to permit data recording on a distributed storage and a mechanism to restrict access to recorded data to authorised organisations only. Over the Blockchain network, the access policy definition for safe data sharing and cryptographic hash content is stored. The real IoT-generated data, on the other hand, is collected via a distributed storage network, which improves availability and security. The proposed scheme’s analysis and performance evaluation show that it is secure and feasible. Furthermore, simulations are undertaken to assess the operating costs of smart contracts and to test the efficacy and viability of the suggested architecture.

Efficient Integrity Auditing Mechanism With Secure Deduplication for Blockchain Storage

Massive nodes in a blockchain form an off-chain distributed storage network to provide storage resources for users to meet large data upload requirements. However, this storage approach introduces security and performance issues. Firstly, it is difficult to guarantee the integrity of the data uploaded, and these data may be easily corrupted or lost. Moreover, uploading excessive duplicate data leads to a waste of storage resources. In this study, to address these issues, with a double-copy storage model for blockchain off-chain storage, a novel public auditing scheme with client-side deduplication is proposed to reduce the storage overhead of nodes and check the integrity of the off-chain data. Based on smart contracts, our scheme could realize efficient user ownership and off-chain data integrity verification automatically. In addition, both data encryption and deduplication are achieved based on message-locked encryption and an improved authenticator generation algorithm. Security analysis and experimental comparisons show that the proposed scheme is effective and practical.

BEHAVIORAL MODEL OF DATA BLOCK RECOVERY SYSTEM FOR THEIR REMOTE DISTRIBUTED STORAGE

A behavioral model of the data block recovery system for their remote distributed storage has been developed. The model was created based on the method of data recovery during their distributed storage in remote storage. The used method allows for restoring information in case of its loss or damages it is possible to recover a completely lost data cluster, provided there are no errors on other clusters, or to recover up to three blocks of data on different clusters. According to the method used, the developed behavioral model provides the functionality of restoring information to the stated requirements. To create a behavioral model, the language of description of integrated circuit hardware was used, namely VHDL. The model provides 5 different devices that combine into one system. During the operation of the model, such functions are performed as solving several systems of equations, sorting, adding two vectors modulo, etc. Simplicity and speed of execution of these functions provide high potential efficiency of execution of the set algorithm. The feasibility of developing a model of this data recovery system is the possible future use of it in distributed storage systems. The advantage is that the model allows you to recover lost data unnoticed by the user, or a data cluster, computer, etc. because it is the developed system must first receive the data stream, identify it, analyze, and if there is no error – just pass it to the user, otherwise, recover data if possible. Data recovery requires the following: data, backup blocks, and a backup matrix, which was created during the initialization of the distributed storage network and stored in the memory of the developed system. In case of impossibility of data recovery, the developed system gives out the corresponding signal.

On Some Capacity-Achieving Fractional Repetition Codes

Modern vehicle-based distributed storage networks need to provide efficient data recovery services due to the high mobility nature of storage vehicles. Compared with the simple replication scheme, erasure codes have been increasingly deployed in practical systems to achieve better storage performance. Specifically, fractional repetition (FR) codes are a class of repair efficient erasure codes characterized by the features of uncoded exact repair and minimum repair bandwidth. In this paper, we consider the capacity of FR codes, which is the maximum amount of information that can be obtained by any user contacting <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$k$</tex-math></inline-formula> out of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$n$</tex-math></inline-formula> storage nodes in the code based system. We present several families of FR codes that achieve a tight upper bound on the FR capacity, which are derived from combinatorial designs. The proposed constructions extend the parameter values of capacity-achieving FR codes to larger sets, and the resulting codes are of practical interest for vehicle-based distributed storage systems.

FileWallet: A File Management System Based on IPFS and Hyperledger Fabric

Online file management systems enable cooperatively editing and sharing. However, due to the cost of communication and storage infrastructures, traditional online file management services, e.g., Google Drive and OneDrive, usually provide limited storage space and relatively low download speed for free users. To achieve better performance, ordinary users have to purchase their expensive services. Moreover, these file management systems are based on centralized architecture and bear the privacy leakage risk, because users’ personal files are stored and controlled by their servers. To address the above problems, we propose a peer-to-peer (P2P) file management system based on IPFS and Hyperledger Fabric, named as FileWallet, which can serve as a personal wallet for individual users or organizations to store and share their files in a secure manner. In FileWallet, the users form a P2P network and a Fabric network, where P2P network builds the connections and distributed storage network and the Fabric network sustains consistent blockchain ledgers to record file operation related transactions. In our FileWallet, the storage and communication costs are mitigated in the decentralized design, and the file owner can fully control the access permission of the file to preserve the file privacy. The design of the system architecture, main functionalities, and system implementations are presented in this paper. The performance of the system is evaluated through experiments, and the experimental results show its wide applicability and scalability.

User Centric Block-Level Attribute Based Encryption in Cloud Using Blockchains

Cloud computing is a collection of distributed storage Network which can provide various services and store the data in the efficient manner. The advantages of cloud computing is its remote access where data can accessed in real time using Remote Method Innovation (RMI). The problem of data security in cloud environment is a major concern since the data can be accessed by any time by any user. Due to the lack of providing the efficient security the cloud computing they fail to achieve higher performance in providing the efficient service. To improve the performance in data security, the block chains are used for securing the data in the cloud environment. However, the traditional block chain technique are not suitable to provide efficient security to the cloud data stored in the cloud. In this paper, an efficient user centric block level Attribute Based Encryption (UCBL-ABE) scheme is presented to provide the efficient security of cloud data in cloud environment. The proposed approach performs data transaction by employing the block chain. The proposed system provides efficient privacy with access control to the user access according to the behavior of cloud user using Data Level Access Trust (DLAT). Based on DLAT, the user access has been restricted in the cloud environment. The proposed protocol is implemented in real time using Java programming language and uses IBM cloud. The implementation results justifies that the proposed system can able to provide efficient security to the data present in and cloud and also enhances the cloud performance.

A Conceptual Architecture in Decentralizing Computing, Storage, and Networking Aspect of IoT Infrastructure

Since the inception of the Internet of Things (IoT), we have adopted centralized architecture for decades. With the vastly growing number of IoT devices and gateways, this architecture struggles to cope with the high demands of state-of-the-art IoT services, which require scalable and responsive infrastructure. In response, decentralization becomes a considerable interest among IoT adopters. Following a similar trajectory, this paper introduces an IoT architecture re-work that enables three spheres of IoT workflows (i.e., computing, storage, and networking) to be run in a distributed manner. In particular, we employ the blockchain and smart contract to provide a secure computing platform. The distributed storage network maintains the saving of IoT raw data and application data. The software-defined networking (SDN) controllers and SDN switches exist in the architecture to provide connectivity across multiple IoT domains. We envision all of those services in the form of separate yet integrated peer-to-peer (P2P) overlay networks, which IoT actors such as IoT domain owners, IoT users, Internet Service Provider (ISP), and government can cultivate. We also present several IoT workflow examples showing how IoT developers can adapt to this new proposed architecture. Based on the presented workflows, the IoT computing can be performed in a trusted and privacy-preserving manner, the IoT storage can be made robust and verifiable, and finally, we can react to the network events automatically and quickly. Our discussions in this paper can be beneficial for many people ranging from academia, industries, and investors that are interested in the future of IoT in general.

Multi-Rack Distributed Data Storage Networks

The majority of works in distributed storage networks assume a simple network model with a collection of identical storage nodes with the same communication cost between the nodes. In this paper, we consider a realistic multi-rack distributed data storage network and present a code design framework for this model. Considering the cheaper data transmission within the racks, our code construction method is able to locally repair the nodes failure within the same rack by using only the survived nodes in the same rack. However, in the case of severe failure patterns when the information content of the survived nodes is not sufficient to repair the failures, other racks will participate in the repair process. By employing the criteria of our multi-rack storage code, we establish a linear programming bound on the size of the code in order to maximize the code rate.

DR-MDS: An Energy-Efficient Coding Scheme in D2D Distributed Storage Network for the Internet of Things

The Internet of Things (IoT) is a promising technology for the collection data among the seamless connectivity with massive physical devices for future 5G networks. While device-to-device (D2D) communication is an emerging technique, which potentially tends to promote the machine-type communications developments of the IoT. In this paper, we consider a wireless distributed storage network where the IoT devices cache decentralized data in each memory and use D2D links to share the whole content, and to save the bandwidth and energy consumptions compared to traditional base station cellular network. We analyze and compare the energy costs of regenerating codes and maximum distance separable (MDS) codes for data downloading and repairing processes with the different parameter configuration. We also define a target named D2D link availability to measure the download and repair efficiency, and obtain the optimal repair intervals of various schemes for the lowest energy cost. Furthermore, we propose a novel caching scheme, termed double replication MDS (DR-MDS), which can modify the repair bandwidth by utilizing more storage occupation, and improve the D2D link availability in the system. The simulation results have shown that the DR-MDS caching significantly outperformed the MDS caching in the energy saving, that also verifies the practicability and the feasibility of our D2D distributed caching system for the low-power IoT scenarios.

Storage and Bandwidth Optimized Reliable Distributed Data Allocation Algorithm

Distributed storage allocation problems are an important optimization problem in reliable distributed storage, which aims to minimize storage cost while maximizing error recovery probability by optimal storage of data in distributed storage nodes. A key characteristic of distributed storage is that data is stored in remote servers across a network. Thus, network resources especially communication links are an expensive and non-trivial resource which should be optimized as well. In this article, the authors present a simulation-based study of the network characteristics of a distributed storage network in the light of several allocation patterns. By varying the allocation patterns, the authors have demonstrated the interdependence between network bandwidth, defined in terms of link capacity and allocation pattern using network throughput as a metric. Motivated by observing the importance of network resource as an important cost metric, the authors have formalized an optimization problem that jointly minimizes both the storage cost and the cost of network resources. A hybrid meta heuristic algorithm is employed that solves this optimization problem by allocating data in a distributed storage system. Experimental results validate the efficacy of the algorithm.

Network-Coding Approach for Information-Centric Networking

The current internet architecture is inefficient in fulfilling the demands of newly emerging internet applications. To address this issue, several over-the-top application-level solutions have been employed, making the overall architecture very complex. Information-centric-networking (ICN) architecture has emerged as a promising alternative solution. The ICN architecture decouples the content from the host at the network level and supports the temporary storage of content in an in-network cache. Fundamentally, the ICN can be considered a multisource, multicast content-delivery solution. Because of the benefits of network coding in multicasting scenarios and proven benefits in distributed storage networks, the network coding is apt for the ICN architecture. In this study, we propose a solvable linear network-coding scheme for the ICN architecture. We also propose a practical implementation of the network-coding scheme for the ICN, particularly for the content-centric network (CCN) architecture, which is termed the coded CCN. The performance results show that the network-coding scheme improves the performance of the CCN and significantly reduces the network traffic and average download delay.

When Can Intelligent Helper Node Selection Improve the Performance of Distributed Storage Networks?

The concept of distributed storage networks (DSNs) mostly follows two main modeling assumptions: Any $k$ out of $n$ surviving nodes should be able to reconstruct the protected file; and if one node fails, the replacement node can access $d$ helper nodes to repair its content either functionally or exactly. Two major existing approaches for DSNs are the so-called regenerating codes (RCs) and locally repairable codes (LRCs), which have different design philosophies and focus on distinct applications. Instead of being limited by the framework of either RCs or LRCs, this work answers a fundamental question for general DSNs: For an arbitrarily given $(n,k,d)$ value, whether there exists an intelligent helper node selection design that can strictly improve the storage-bandwidth tradeoff when compared with naive blind helper selection. Surprisingly, the answer is negative for a large set of $(n,k,d)$ values. Namely, for those $(n,k,d)$ values even the best helper selection design offers no gain over a blind solution. We call those $(n,k,d)$ values indifferent-to-helper-selection (ITHS). The main contribution of this work is a necessary and sufficient condition that characterizes whether an $(n,k,d)$ value is ITHS. As a fundamental study, this work assumes functional repair with unlimited computing power for encoding/decoding and focuses on the fundamental performance limits of intelligent helper selection. A new helper selection scheme, termed family helper selection , is proposed and used in the achievability analysis. For some scenarios, the proposed scheme is indeed optimal (as good as any helper selection one can design).

Generalized Pyramid Codes for Versioned Data in Distributed Storage Networks

A distributed storage framework called sparsity exploiting coding (SEC) was recently proposed for storing multiple versions of data object, wherein the objects are stored in a differential manner in order to reduce the I/O reads when retrieving multiple versions. It was shown that the design of erasure codes for the SEC framework requires Cauchy-matrix-based maximum distance separable (MDS) codes, which, however, do not enjoy the locally repairable property during the repair process. In this letter, we study the suitability of generalized pyramid codes (GPCs), a well-known class of locally repairable codes, to the SEC framework, and analyze their capability to retrieve multiple versions with few I/O reads. We show that GPCs can retrieve correlated multiple versions with fewer I/O reads than the overall object size, however, requiring more I/O reads than that by Cauchy-matrix-based MDS codes. This penalty in the I/O reads comes at the advantage of availing the locally repairable property in the SEC framework. We also show that GPCs require fewer I/O reads than the optimal locally repairable codes, in some cases.

Synchronization and Deduplication in Coded Distributed Storage Networks

We consider the problem of synchronizing coded data in distributed storage networks undergoing insertion and deletion edits. We present modifications of distributed storage codes that allow updates in the parity-check values to be performed with one round of communication at low bit rates and with small storage overhead. Our main contributions are novel protocols for synchronizing frequently updated and semi-static data based on functional intermediary coding involving permutation and Vandermonde matrices.

Guest Editorial Channel Modeling, Coding and Signal Processing for Novel Physical Memory Devices and Systems

The digital universe is doubling every two years and expected to reach an unwieldy 44 zettabytes into the next decade. To cope with the ever increasing need for storing, transmitting and retrieving huge amounts of data, cloud storage, data centers and other massively distributed storage networks have emerged. These rely on efficient memory technologies at the physical level for speed, reliability and energy efficiency.

Efficiently sphere-decodable physical layer transmission schemes for wireless storage networks

Three transmission schemes over a new type of multiple-access channel (MAC) model with inter-source communication links are proposed and investigated in this paper. This new channel model is well motivated by, e.g., wireless distributed storage networks, where communication to repair a lost node takes place from helper nodes to a repairing node over a wireless channel. Since in many wireless networks nodes can come and go in an arbitrary manner, there must be an inherent capability of inter-node communication between every pair of nodes. Assuming that communication is possible between every pair of helper nodes, the newly proposed schemes are based on various smart time-sharing and relaying strategies. In other words, certain helper nodes will be regarded as relays, thereby converting the conventional uncooperative multiple-access channel to a multiple-access relay channel (MARC). The diversity-multiplexing gain tradeoff (DMT) of the system together with efficient sphere-decodability and low structural complexity in terms of the number of antennas required at each end is used as the main design objectives. While the optimal DMT for the new channel model is fully open, it is shown that the proposed schemes outperform the DMT of the simple time-sharing protocol and, in some cases, even the optimal uncooperative MAC DMT. While using a wireless distributed storage network as a motivating example throughout the paper, the MAC transmission techniques proposed here are completely general and as such applicable to any MAC communication with inter-source communication links.

Distributed Information System for Processing and Storage of Meteorological Data

The following article features the results of developing distributed network storage of ground meteorological observation data. The data is represented with the national variant of international rapid transmission code of environment data from meteorological stations across the Russian Federation. They are available for researchers in both visual and common export formats. The design of the distributed network storage of meteorological data includes the following modules: dispatcher module (monitors calculation nodes, distributes data to nodes, processes client requests), client module (allows external researchers to access the meteorological data), terminal module (used to import new meteorological data), data processing and storage module (a node for distributed meteorological data storage, consists of two sub-modules for data processing and data storage respectively). The article displays the results of practical testing of the developed software. To simulate the cluster of informational and calculation servers in the pilot project, multithreading was used. Multithreading is supported by nearly every operational system for parallel data processing. The development tools chosen for the network storage allowed to design storage module interaction with the optimal efficiency, to ensure proper performance, stability and reliability of processing and managing large amounts of data. The obtained results allow using the designs for efficient management of meteorological surface observation data, for rapid data gathering, for systematization and storage of hydro-meteorological data in different alphanumeric codes and other related categories.

Guest Editorial Communication Methodologies for the Next-Generation Storage Systems

This issue consists of 22 high-caliber papers with contributions from both academia and industry. The papers are organized into the following six sections: (i) Channel Modeling and Signal Processing Algorithms for Emerging Memory Technologies, (ii) Error Control Coding Techniques for Flash Memories, (iii) Algebraic Methods with Applications to Non- Volatile Memories, (iv) Polar Codes with Application to Storage, (v) Performance Limits of Storage Systems, and (vi)Codes for Distributed Network Storage.