Storage Codes Research Articles

Single-Server Private Information Retrieval Schemes are Equivalent to Locally Recoverable Coding Schemes

The Private Information Retrieval (PIR) problem has recently attracted a significant interest in the information-theory community. In this problem, a client wants to download one or more messages belonging to a database while protecting the identity of the downloaded message(s). In this article, we focus on the scenarios in which (i) the entire database is stored on a single server and (ii) the client has prior side information, namely a subset of messages unknown to the server. Such prior side information is necessary to enable efficient private information retrieval in the single server scenario. In the last decade, there has also been a significant interest in Locally Recoverable Codes (LRCs), a class of storage codes in which each symbol can be recovered from a limited number of other symbols. More recently, there is an interest in cooperative locally recoverable codes, i.e., codes in which multiple symbols can be recovered from a small set of other code symbols. The central problem in this context is given a set of code parameters to design an LRC scheme that includes a locally recoverable code along with encoding, decoding, and repair functions. The paper establishes an equivalence between the single-server PIR schemes and LRC schemes. In particular, we present explicit algorithms that transform a given PIR scheme into an LRC scheme and vice versa. We show that (i) PIR schemes for retrieving a single message are equivalent to classical LRC schemes; and (ii) PIR schemes for retrieving multiple messages are equivalent to cooperative LRC schemes. These equivalence results allow us to recover upper bounds on the download rate for PIR schemes, and to obtain a novel rate upper bound on cooperative LRC schemes. Our results cover schemes based on both linear and non-linear codes.

Private Information Retrieval Schemes With Product-Matrix MBR Codes

A private information retrieval (PIR) scheme allows a user to retrieve a file from a database without revealing any information on the file being requested. As of now, PIR schemes have been proposed for several kinds of storage systems, including replicated and MDS-coded systems. However, the problem of constructing PIR schemes on regenerating codes has been sparsely considered. A regenerating code is a storage code whose codewords are distributed among nodes, enabling efficient storage of files, as well as low-bandwidth retrieval of files and repair of nodes. Minimum-bandwidth regenerating (MBR) codes define a family of regenerating codes allowing a node repair with optimal bandwidth. Rashmi, Shah, and Kumar obtained a large family of MBR codes using the product-matrix (PM) construction. In this work, a new PIR scheme over PM-MBR codes is designed. The inherent redundancy of the PM structure is used to reduce the download communication complexity of the scheme. A lower bound on the PIR capacity of MBR-coded PIR schemes is derived, showing an interesting storage space vs. PIR rate trade-off compared to existing PIR schemes with the same reconstruction capability. The present scheme also outperforms a recent PM-MBR PIR construction of Dorkson and Ng.

On the Storage Cost of Private Information Retrieval

We consider the fundamental tradeoff between the storage cost and the download cost in private information retrieval systems, without any explicit structural restrictions on the storage codes, such as maximum distance separable codes or uncoded storage. Two novel outer bounds are provided, which have the following implications. When the messages are stored without any redundancy across the databases, the optimal PIR strategy is to download all the messages; on the other hand, for PIR capacity-achieving codes, each database can reduce the storage cost, from storing all the messages, by no more than one message on average. We then focus on the two-message two-database case, and show that a stronger outer bound can be derived through a novel pseudo-message technique. This stronger outer bound suggests that a precise characterization of the storage-download tradeoff may require non-Shannon type inequalities, or at least more sophisticated bounding techniques.

An Intelligent Optimization Algorithm for Constructing a DNA Storage Code: NOL-HHO.

The high density, large capacity, and long-term stability of DNA molecules make them an emerging storage medium that is especially suitable for the long-term storage of large datasets. The DNA sequences used in storage need to consider relevant constraints to avoid nonspecific hybridization reactions, such as the No-runlength constraint, GC-content, and the Hamming distance. In this work, a new nonlinear control parameter strategy and a random opposition-based learning strategy were used to improve the Harris hawks optimization algorithm (for the improved algorithm NOL-HHO) in order to prevent it from falling into local optima. Experimental testing was performed on 23 widely used benchmark functions, and the proposed algorithm was used to obtain better coding lower bounds for DNA storage. The results show that our algorithm can better maintain a smooth transition between exploration and exploitation and has stronger global exploration capabilities as compared with other algorithms. At the same time, the improvement of the lower bound directly affects the storage capacity and code rate, which promotes the further development of DNA storage technology.

Achieving Load-Balanced, Redundancy-Free Cluster Caching with Selective Partition

Data-intensive clusters increasingly rely on in-memory storages to improve I/O performance. However, the routinely observed file popularity skew and load imbalance create hot spots, which significantly degrade the benefits of in- memory caching. Common approaches to tame load imbalance include copying multiple replicas of hot files and creating parity chunks using storage codes. Yet, these techniques either suffer from high memory overhead due to cache redundancy or incur non-trivial encoding/decoding complexity. In this paper, we propose an effective approach to achieve load balancing without cache redundancy or encoding/decoding overhead. Our solution, termed SP-Cache, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">selectively partitions</i> files based on the loads they contribute and evenly caches those partitions across the cluster. We develop an efficient algorithm to determine the optimal number of partitions for a hot file—too few partitions are incapable of mitigating hot spots, while too many are susceptible to stragglers. We have implemented SP-Cache atop Alluxio, a popular in-memory distributed storage system, and evaluated its performance through EC2 deployment and trace-driven simulations. SP-Cache can quickly react to the changing load by dynamically re-balancing cache servers. Compared to the state-of-the-art solution, SP-Cache reduces the file access latency by up to 40 percent in both the mean and the tail, using 40 percent less memory.

Performance Analysis on Distributed Storage Systems in Ring Networks

Distributed storage systems can provide reliable service of data storage. Lots of scenarios, such as data center, P2P network are all applications of distributed storage systems. There are mainly three storage schemes for distributed storage systems, including replication , erasure codes and regenerating codes [2] . With the same storage space, more reliable data storage is assured by erasure codes than replication . However, replication can save more repair bandwidth (i.e., the communication cost for node repair) than erasure codes . To ensure both high reliability and low repair bandwidth, regenerating codes are intensively investigated. In practical systems, the communication between two nodes may be restricted by physical factors or economic considerations, such as transmission distance or network traffic. In this paper, we propose a novel model considering the connectivity limitations in ring networks. Each node can only connect to or communicate with its neighbors within a given range. Inspired by the concept of regenerating codes , we investigate the theoretical necessary condition for all possible storage codes. Then, a tradeoff between storage space and repair bandwidth is derived by analyzing the information flow graph of our model. Two interesting points on the tradeoff curve are further investigated, corresponding to the ring minimum-storage-regenerating (rMSR) codes, and the ring minimum-bandwidth-regenerating (rMBR) codes. Besides, two types of code constructions are presented. (i) For both rMSR and rMBR codes, we provide code constructions under exact repair models. (ii) Under functional repair models, we derive the size of the coding finite field, based on which the algorithm in [3] can be applied to construct the corresponding codes. And finally, the superiority of our codes is validated by both theoretical and simulation results. Simulations show that our codes outperform baseline schemes, and the reduction on repair bandwidth can be up to 66% of the original data size.

Fluorescent materials-based information storage

Fluorescent materials-based information storage systems are characterised by attractive data security, stability, and stimuli responsive features. We summarise here, recent progress in the area of fluorescent materials-based information storage codes.

Distributed Storage Codes Based on Double- Layered Piggybacking Framework

Piggybacking codes are a class of distributed storage codes adopted in distributed storage systems, which received much attention in recent years. Two novel piggybacking constructions based on the double-layered piggybacking framework are proposed. They can reduce the repair bandwidth of both the systematic nodes and the parity nodes efficiently. The average repair bandwidth ratio for the two constructions approaches to zero asymptotically as the number of systematic nodes and parity nodes tends to infinity. Compared with other piggybacking codes, the two proposed double-layered piggybacking codes, especially the second one, not only require less design constraints, but also obtain the optimal comprehensive repair efficiency, which further save the amount of data read and downloaded.

Cognitive and artificial intelligence system for logistics industry

Artificial intelligence is an important branch of cognitive science. Artificial intelligence has been applied to medical, education, logistics and other industries, which has broad prospects for development. Hotel logistics industry uses artificial intelligence technology to complete intelligent search, face recognition, combined with large data calculation and planning reasonable path in warehousing. Taking China's hotel logistics industry as the research object, this paper analyses the application of artificial intelligence technology in the logistics industry. In the warehousing process, artificial intelligence technologies including compile storage code, automatic picking with automated guided vehicle, warehouse robot to improve work efficiency. Logistics distribution links use artificial intelligence technology to plan the best path, improve the recognition rate of express waybill that save a lot of labour. Artificial intelligence technology allocates logistics resources, optimises logistics links, and improves logistics efficiency and other measures to promote the development of logistics informatisation and automation.

K-Means Multi-Verse Optimizer (KMVO) Algorithm to Construct DNA Storage Codes

In an era of information explosion, dealing with massive data has become a problem. Since DeoxyriboNucleic Acid (DNA) is a high-density storage medium with long storage endurance, a DNA based storage system is a viable solution. The first consideration of a DNA storage system is the DNA codes, which can avoid non-specific hybridization of DNA strands in the hybridization reaction process by using related constraints, such as Hamming distance constraints, GC-content constraints, and no-runlength constraints. A K-means Multi-Verse Optimizer (KMVO) algorithm is proposed to construct a better code boundary than the previous Multi-Verse Optimizer (MVO) algorithm that satisfies the above constraints. Our results can store information more efficiently over a given length, increasing storage utilization.

Multi-Rack Distributed Data Storage Networks

The majority of works in distributed storage networks assume a simple network model with a collection of identical storage nodes with the same communication cost between the nodes. In this paper, we consider a realistic multi-rack distributed data storage network and present a code design framework for this model. Considering the cheaper data transmission within the racks, our code construction method is able to locally repair the nodes failure within the same rack by using only the survived nodes in the same rack. However, in the case of severe failure patterns when the information content of the survived nodes is not sufficient to repair the failures, other racks will participate in the repair process. By employing the criteria of our multi-rack storage code, we establish a linear programming bound on the size of the code in order to maximize the code rate.

Linear programming bounds for distributed storage codes

A major issue of locally repairable codes is their robustness. If a local repair group is not able to perform the repair process, this will result in increasing the repair cost. Therefore, it is critical for a locally repairable code to have multiple repair groups. In this paper we consider robust locally repairable coding schemes which guarantee that there exist multiple distinct (not necessarily disjoint) alternative local repair groups for any single failure such that the failed node can still be repaired locally even if some of the repair groups are not available. We use linear programming techniques to establish upper bounds on the size of these codes. We also provide two examples of robust locally repairable codes that are optimal regarding our linear programming bound. Furthermore, we address the update efficiency problem of the distributed data storage networks. Any modification on the stored data will result in updating the content of the storage nodes. Therefore, it is essential to minimise the number of nodes which need to be updated by any change in the stored data. We characterise the update-efficient storage code properties and establish the necessary conditions of existence update-efficient locally repairable storage codes.

A new piggybacking design for systematic MDS storage codes

Distributed storage codes have important applications in the design of modern storage systems. In a distributed storage system, every storage node has a probability to fail and once an individual storage node fails, it must be reconstructed using data stored in the surviving nodes. Computation load and network bandwidth are two important issues we need to concern when repairing a failed node. The traditional maximal distance separable (MDS) storage codes have low repair complexity but high repair bandwidth. On the contrary, minimal storage regenerating (MSR) codes have low repair bandwidth but high repair complexity. Fortunately, the newly introduced piggyback codes combine the advantages of both ones. In this paper, by introducing a novel piggybacking design framework for systematic MDS codes, we construct a storage code whose average repair bandwidth rate, i.e., the ratio of average repair bandwidth and the amount of the original data, can be as low as $\frac{\sqrt{2r-1}}{r}$, which significantly improves the ratio $\frac{r-1}{2r-1}$ of the previous result. In the meanwhile, every failed systematic node of the new code can be reconstructed quickly using the decoding algorithm of an MDS code, only with some additional additions over the underlying finite field. This is very fast compared with the complex matrix multiplications needed in the repair of a failed node of an MSR code.

Functional repair codes: a view from projective geometry

Storage codes are used to ensure reliable storage of data in distributed systems; functional repair codes have the additional property that individual storage nodes that fail may be repaired efficiently, preserving the ability to recover original data and to further repair failed nodes. In this paper we show that the existing predominant coding theoretic and vector space models of repair codes can be given a unified treatment in a projective geometric framework, which permits a natural treatment of results such as the cutset bound. We find that many of the constructions proposed in the literature can be seen to arise from well-studied geometric objects, and that this perspective provides opportunities for generalisations and new constructions that can lead to greater flexibility in trade-offs between various desirable properties. We use this framework to explore the notion of strictly functional repair codes, for which there exist nodes that cannot be replaced exactly, and discuss how strict functionality can arise. We also consider the issue that the view of a repair code as a collection of sets of vector/projective subspaces is recursive in nature and makes it hard to discern when a collection of nodes forms a repair code. We provide another view using directed graphs that gives us non-recursive criteria for determining whether a family of collections of subspaces constitutes a functional, exact, or strictly functional repair code, which may be of use in searching for new codes with desirable properties.

Private Information Retrieval From Coded Storage Systems With Colluding, Byzantine, and Unresponsive Servers

The problem of private information retrieval (PIR) from coded storage systems with colluding, Byzantine, and unresponsive servers is considered. An explicit scheme using an $[n,k]$ Reed–Solomon storage code is designed, protecting against $t$ -collusion, and handling up to $b$ Byzantine and $r$ unresponsive servers, when $n>k+t+2b+r-1$ . This scheme achieves a PIR rate of $(({n-r-(k+2b+t-1)})/{n-r})$ . In the case where the capacity is known, namely, when $k=1$ , it is asymptotically capacity achieving as the number of files grows. Finally, the scheme is adapted to symmetric PIR.

A general private information retrieval scheme for MDS coded databases with colluding servers

The problem of private information retrieval (PIR) gets renewed attentions in recent years due to its information-theoretic reformulation and applications in distributed storage systems. Let M files be stored in a distributed storage system consisting of N servers, where each file is stored via an (N, K)-MDS code. A PIR scheme will allow a user to retrieve a specific file from the distributed storage system without revealing the identity of the file to any T colluding servers. PIR rate is defined as the number of symbols privately retrieved per one downloaded symbol and the supremum of all achievable rates is called the PIR capacity. The capacity has been solved for some degenerate cases, i.e. $$K=1$$ or $$T=1$$ . For the general case $$K\ge 2$$ and $$T\ge 2$$ , the exact PIR capacity remains unknown. In this paper we propose a general private information retrieval scheme for MDS coded databases with colluding servers achieving PIR rate $$(1+R+R^2+\cdots +R^{M-1})$$ , where $$R=1-\frac{{{N-T}\atopwithdelims ()K}}{{N\atopwithdelims ()K}}$$ . Our scheme captures the essence of the optimal schemes for degenerate cases. We also compare our scheme with some other known PIR schemes for non-degenerate cases. The advantages of our scheme include its independence of the property of the storage code and a better performance when the number of files M is small.

Sparks Are Flying: The Growing Tension Between Energy Storage and Fire Safety Stakeholders

This paper explores how overly zealous and protective battery energy storage codes could ultimately work against the goals of consumers by demonstrating the complex relationship between energy needs, potential hazards, and regulation. To ground this discussion, this paper first gives an overview of what energy storage is, what the key hazards are regarding storage, and why it is vital component of renewable energy generation. The nuanced relationship between the needs of consumers and motivation of regulators is illustrated by examining how energy storage could have mitigated some of Hurricane Sandy’s impact on New York City and also how N.Y.C.’s 2018 outdoor energy storage guidelines effectively limit the access to energy storage. Finally, this paper provides several recommendations regarding how to develop battery energy storage regulations that better protect consumers while still enabling consumers to benefit from energy storage.

Symmetric Private Information Retrieval from MDS Coded Distributed Storage With Non-Colluding and Colluding Servers

A user wants to retrieve a file from a database without revealing the identity of the file retrieved to the operator of the database (server), which is known as the problem of private information retrieval (PIR). If it is further required that the user obtains no information about the other files in the database, the concept of symmetric PIR (SPIR) is introduced to guarantee privacy for both parties. For SPIR, the server(s) need to access some randomness independent of the database, to protect the content of undesired files from the user. The information-theoretic capacity of SPIR is defined as the maximum number of information bits of the desired file retrieved per downloaded bit. In this paper, the problem of SPIR is studied for a distributed storage system with $N$ servers (nodes), where all data (including the files and the randomness) are stored in a distributed way. Specifically, the files are stored by an $(N,K_{C})$ -MDS storage code. The randomness is distributedly stored such that any $K_{C}$ servers store independent randomness information. We consider two scenarios regarding to the ability of the storage nodes to cooperate. In the first scenario considered, the storage nodes do not communicate or collude. It is shown that the SPIR capacity for MDS-coded storage (hence called MDS-SPIR) is $1-\frac {K_{C}}{N}$ , when the amount of the total randomness of distributed nodes (unavailable at the user) is at least $\frac {K_{C}}{N - K_{\vphantom {R_{l}}C}}$ times the file size. Otherwise, the MDS-SPIR capacity equals zero. The second scenario considered is the $T$ -colluding SPIR problem (hence called TSPIR). Specifically, any $T$ out of $N$ servers may collude, that is, they may communicate their interactions with the user to guess the identity of the requested file. In the special case with $K_{C}=1$ , i.e., the database is replicated at each node, the capacity of TSPIR is shown to be $1-\frac {T}{N}$ , with the ratio of the total randomness size relative to the file size be at least $\frac {T}{\vphantom {R_{l}}N - T}$ . For TSPIR with MDS-coded storage (called MDS-TSPIR for short), when restricted to schemes with additive randomness where the servers add the randomness to the answers regardless of the queries received, the capacity is proved to equal $1-\frac {K_{C} + T - 1}{N}$ , with total randomness at least $\frac {K_{C} + T - 1}{N - K_{\vphantom {R_{l}}C} - T + 1}$ times the file size. The MDS-TSPIR capacity for general schemes remains an open problem.

Communication Cost for Updating Linear Functions When Message Updates are Sparse: Connections to Maximally Recoverable Codes

We consider a communication problem in which an update of the source message needs to be conveyed to one or more distant receivers that are interested in maintaining specific linear functions of the source message. The setting is one in which the updates are sparse in nature, and where neither the source nor the receiver(s) is aware of the exact {\em difference vector}, but only know the amount of sparsity that is present in the difference-vector. Under this setting, we are interested in devising linear encoding and decoding schemes that minimize the communication cost involved. We show that the optimal solution to this problem is closely related to the notion of maximally recoverable codes (MRCs), which were originally introduced in the context of coding for storage systems. In the context of storage, MRCs guarantee optimal erasure protection when the system is partially constrained to have local parity relations among the storage nodes. In our problem, we show that optimal solutions exist if and only if MRCs of certain kind (identified by the desired linear functions) exist. We consider point-to-point and broadcast versions of the problem, and identify connections to MRCs under both these settings. For the point-to-point setting, we show that our linear-encoder based achievable scheme is optimal even when non-linear encoding is permitted. The theory is illustrated in the context of updating erasure coded storage nodes. We present examples based on modern storage codes such as the minimum bandwidth regenerating codes.

The Capacity of Private Information Retrieval From Coded Databases

We consider the problem of private information retrieval (PIR) over a distributed storage system. The storage system consists of $N$ non-colluding databases, each storing an MDS-coded version of $M$ messages. In the PIR problem, the user wishes to retrieve one of the available messages without revealing the message identity to any individual database. We derive the information-theoretic capacity of this problem, which is defined as the maximum number of bits of the desired message that can be privately retrieved per one bit of downloaded information. We show that the PIR capacity in this case is $C=(1+{K}/{N}+{K^{2}}/{N^{2}}+\cdots +{K^{M-1}}/{N^{M-1}})^{-1}=(1+R_{c}+R_{c}^{2}+\cdots +R_{c}^{M-1})^{-1}=({1-R_{c}})/({1-R_{c}^{M}})$ , where $R_{c}$ is the rate of the $(N,K)$ MDS code used. The capacity is a function of the code rate and the number of messages only regardless of the explicit structure of the storage code. The result implies a fundamental tradeoff between the optimal retrieval cost and the storage cost when the storage code is restricted to the class of MDS codes. The result generalizes the achievability and converse results for the classical PIR with replicated databases to the case of MDS-coded databases.