Distributed Storage Systems Research Articles

A Blockchain assisted fog computing for secure distributed storage system for IoT Applications

A Blockchain assisted fog computing for secure distributed storage system for IoT Applications

Enhancing the Efficiency of Distributed Storage Systems through XOR-Optimized Cauchy Reed-Solomon Codes

In today‘s society, we have entered the era of big data and the Internet of Things. The rise of services such as artificial intelligence, cloud storage, and the metaverse has led to increased demands for data transmission and storage. People now require not only greater capacity but also higher efficiency, more powerful hardware, and advanced algorithms. storageThe Reed-Solomon (RS) code, as a highly effective error correction code, is officially applied in fields such as data storage and data transmission. The Cauchy Reed-Solomon (CRS) Code, which is based on the RS code, utilizes a Cauchy matrix instead of the original Vandermonde matrix and replaces the original matrix multiplication with XOR operations. This paper primarily explores how the XOR-optimized Cauchy Reed-Solomon Code improves the efficiency of distributed storage systems, It was discovered that using lookup tables to store frequently used data and operations can further enhance the efficiency of storage systems and its future applications.

Large-scale energy storage for carbon neutrality: thermal energy storage for electrical vehicles

AbstractThermal Energy Storage (TES) systems are pivotal in advancing net-zero energy transitions, particularly in the energy sector, which is a major contributor to climate change due to carbon emissions. In electrical vehicles (EVs), TES systems enhance battery performance and regulate cabin temperatures, thus improving energy efficiency and extending vehicle range. The enhanced efficiency reduces overall energy consumption in EVs. Consequently, this reduction in energy demand can lead to decreased infrastructure needs, minimising the scale and investment required in energy production and distribution systems. Furthermore, the integration of TES with existing infrastructure allows for the simultaneous charging of thermal and electrical energy, leveraging waste heat or renewable energy sources. This not only cuts costs by optimizing resource use but also bolsters sustainability by minimising reliance on non-renewable energy sources. The widespread adoption of TES in EVs could transform these vehicles into nodes within large-scale, distributed energy storage systems, thus supporting smart grid operations and enhancing energy security. Strategic investments and regulatory updates are essential to realise a sustainable, carbon-neutral transportation future, underpinned by robust, cost-efficient infrastructure.

Optimization and performance improvement of distributed data storage in hybrid storage systems

With the rapid development of information technology, the storage and processing of massive data has become one of the important challenges facing the current computing field. Traditional centralized storage systems have been unable to meet the needs of big data applications, while distributed storage systems have become the infrastructure of modern data centers because of their high scalability and high availability. However, in practical applications, a single distributed storage model is often difficult to balance cost-effectiveness and performance requirements. Therefore, this paper proposes a hybrid storage system model combining local storage and cloud storage resources and discusses a series of optimization strategies for this model. The focus of the research is to improve the overall performance of the system through the design of intelligent data sharding, redundancy and fault tolerance mechanisms, and the application of effective load balancing technology. The effectiveness and superiority of the proposed method are verified by testing and analyzing several typical application scenarios on the experimental platform. The experimental results show that the optimized hybrid storage system not only significantly improves the speed of data access, but also effectively reduces the storage cost, demonstrating its potential in future large-scale data management.

DISTRIBUTED HIGH-PERFORMANCE COMPUTING METHODS FOR ACCELERATING DEEP LEARNING TRAINING

This paper comprehensively analyzes distributed high-performance computing methods for accelerating deep learning training. We explore the evolution of distributed computing architectures, including data parallelism, model parallelism, and pipeline parallelism, and their hybrid implementations. The study delves into optimization techniques crucial for large-scale training, such as distributed optimization algorithms, gradient compression, and adaptive learning rate methods. We investigate communication-efficient algorithms, including Ring All Reduce variants and decentralized training approaches, which address the scalability challenges in distributed systems. The research examines hardware acceleration and specialized systems, focusing on GPU clusters, custom AI accelerators, high-performance interconnects, and distributed storage systems optimized for deep learning workloads. Finally, we discuss this field's challenges and future directions, including scalability-efficiency trade-offs, fault tolerance, energy efficiency in large-scale training, and emerging trends like federated learning and neuromorphic computing. Our findings highlight the synergy between advanced algorithms, specialized hardware, and optimized system designs in pushing the boundaries of large-scale deep learning, paving the way for future breakthroughs in artificial intelligence.

An Evaluation of DAOS for Simulation and Deep Learning HPCWorkloads

Traditionally, distributed storage systems have relied upon the interfaces provided by OS kernels to interact with storage hardware. However, much research has shown that OSes impose serious overheads on every I/O operation, especially on high-performance storage and networking hardware (e.g., PMEM and 200GBe). Thus, distributed storage stacks are being re-designed to take advantage of this modern hardware by utilizing new hardware interfaces which bypass the kernel entirely. However, the impact of these optimizations have not been well-studied for real HPC workloads on real hardware. In this work, we provide a comprehensive evaluation of DAOS: a state-of-the-art distributed storage system which re-architects the storage stack from scratch for modern hardware.We compare DAOS against traditional storage stacks and demonstrate that by utilizing optimal interfaces to hardware, performance improvements of up to 6x can be observed in real scientific applications.

An efficient blockchain-based framework for file sharing

File sharing, being the foundation of the Internet, has traditionally relied on a centralized service architecture resulting in significant maintenance costs. Moreover, due to the lack of an effective file management system, instances of sensitive information going out of control and loss of confidentiality in file sharing have occurred frequently. In order to address the difficulty of tamper detection and the lack of supervision in the entire process of file transfer in the current Internet environment, this paper designs a blockchain-based system architecture for secure sharing of electronic documents. An efficient blockchain model is used in our framework, and with the help of distributed storage system and asymmetric encryption technology, file sharing can be controlled, reliable and traceable in the transfer process. Referring to existing consensus mechanisms, e.g., Delegated Proof of Stake (DPoS) and Practical Byzantine Fault Tolerance (PBFT), we propose a new consensus for efficient and secure file sharing. Our experimental results show that our framework can maintain a higher throughput than existing schemes

Enhancing Hadoop distributed storage efficiency using multi-agent systems

<p>Distributed storage systems play a pivotal role in modern data-intensive applications, with Hadoop distributed file system (HDFS) being a prominent example. However, optimizing the efficiency of such systems remains a complex challenge. This research paper presents a novel approach to enhance the efficiency of distributed storage by leveraging multi-agent systems (MAS). Our research is centered on enhancing the efficiency of the HDFS by incorporating intelligent agents that can dynamically assign storage tasks to nodes based on their performance characteristics. Utilizing a decentralized decision-making framework, the suggested approach based on MAS considers the real-time performance of nodes and allocates storage tasks adaptively. This strategy aims to alleviate performance bottlenecks and minimize data transfer latency. Through extensive experimental evaluation, we demonstrate the effectiveness of our approach in improving HDFS performance in terms of data storage, retrieval, and overall system efficiency. The results reveal significant reductions in job execution times and enhanced resource utilization, there by offering a promising avenue for enhancing the efficiency of distributed storage systems.</p>

Flimm: Foreground traffic aware data migration manager for distributed storage system

Currently, decentralized distributed storage system is widely used in various scenarios, because decentralized data placement algorithms observably improve scalability and robustness of distributed storage system. However, storage system applied with decentralized data placement algorithm suffers from uncontrollable data migration during expanding the clusters, which lead to dramatic fluctuation of performance. To maintain reliability and stability of storage system, administrators are under great pressure to migrate data during expansion or contraction. Not only the urgent time constraints to expansion but also low interference to foreground application are required during data migration period. It is a difficult challenge for in-experienced administrators to tackle.To address this challenge, this paper introduces Flimm, a fast and low-interference data migration manager that is aware of storage resource utilization and performs data migration with low interference to foreground applications. Flimm mitigates interference by scheduling a subset of migration tasks with careful consideration. Moreover, Flimm adapts to varying foreground application workloads through adaptive migration speed control, providing administrators with convenient and secure parameters for tuning. In our experiments, Flimm demonstrated a 1.2x acceleration in migration speed and achieved a 20% reduction in interference.

Symmetrical Data Recovery: FPGA-Based Multi-Dimensional Elastic Recovery Acceleration for Multiple Block Failures in Ceph Systems

In the realm of Ceph distributed storage systems, ensuring swift and symmetrical data recovery during severe data corruption scenarios is pivotal for data reliability and system stability. This paper introduces an innovative FPGA-based Multi-Dimensional Elastic Recovery Acceleration method, termed AMDER-Ceph. Utilizing FPGA technology, this method is a pioneer in accelerating erasure code data recovery within such systems symmetrically. By harnessing the parallel computing power of FPGAs and optimizing Cauchy matrix binary operations, AMDER-Ceph significantly enhances data recovery speed and efficiency symmetrically. Our evaluations in real-world Ceph environments show that AMDER-Ceph achieves up to 4.84 times faster performance compared with traditional methods, especially evident in the standard 4 MB block size configurations of Ceph systems.

Correction: Accelerating erasure coding by exploiting multiple repair paths in distributed storage systems

Correction: Accelerating erasure coding by exploiting multiple repair paths in distributed storage systems

On the Optimized Management of Activated Distributed Storage Systems: A Novel Approach to Flood Mitigation

New flood records are being set across the world as precipitation patterns change due to a warming climate. Despite the presence of longstanding water management infrastructure like levees and reservoirs, this rise in flooding has been met with property damage, loss of life, and hundreds of billions in economic impact, suggesting the need for new solutions. In this work, the authors suggest the active management of distributed networks of ponds, wetlands and retention basins that already exist across watersheds for the mitigation of flood damages. As an example of this approach, we investigate optimal control of the gated outlets of 130 such locations within a small watershed using linear programming, genetic algorithms, and particle swarm optimization, with the objective of reducing downstream flow and maximizing basin storage. When compared with passive operation (i.e., no gated outlets) and a uniformly applied active management scheme designed to store water during heavy rainfall, the optimal control techniques (1) reduce the magnitudes of peak flow events by up to 10%, (2) reduce the duration of flood crests for up to several days, and (3) preserve additional storage across the watershed for future rainfall events when compared with active management. Combined, these findings provide both a better understanding of dynamically controlled distributed storage as a flood fighting technique and a springboard for future work aimed at its use for reducing flood impacts.

АНАЛІЗ ПОТОЧНОГО СТАНУ ЕНЕРГОСИСТЕМИ ТА СТРАТЕГІЇ РОЗВИТКУ РОЗОСЕРЕДЖЕНОЇ ГЕНЕРАЦІЇ В УКРАЇНІ

The article analyzes the state of the Unified Energy System (UES) of Ukraine as of the beginning of 2023. The main focus of the article is on the issue of energy security and the consequences for the UES of Ukraine after mass attacks on energy facilities from the beginning of a full-scale invasion until the beginning of 2023. The assessment of the main shortcomings of the current model of the operation of the power system was carried out and the current short-term strategies for its modernization were considered in order to prepare for the heating season of 2023-2024. During the analysis of potentially possible short-term strategies, the main attention was paid to the recovery plan of Ukraine, available capacities of the Gas Transport System (GTS) of Ukraine, potential of using "Floating Power Plants", poten-tial possibilities of the operation of distribution networks (DM) in the "island mode" and stimulating an increase in the share of distributed sources generation (DSG) and energy storage systems (ESS). The current results of the implemen-tation of the above-mentioned strategies and the potential of their development after the end of a full-scale war in ac-cordance with the energy strategy of Ukraine until 2035 and the main theses of the energy strategy of Ukraine until 2050 were presented. Based on the results of the analysis, the recommendation regarding the implementation of the preparation of the UES of Ukraine for the next heating season were formed. Bibl. 10, fig. 2.

Constacyclic locally recoverable codes from their duals

A code has locality r if a symbol in any coordinate of a codeword in the code can be recovered by accessing the value of at most r other coordinates. Such codes are called locally recoverable codes (LRCs for short). Since LRCs can recover a failed node by accessing the minimum number of the surviving nodes, these codes are used in distributed storage systems such as Microsoft Azure. In this paper, constacyclic LRCs are obtained from their parity-check polynomials. Constacyclic codes with locality r≤2\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$r\\le 2$$\\end{document} and dimension 2 are obtained and a sufficient and necessary condition for these codes to have locality 1 is given. Then, the construction is generalized. Distance optimal constacyclic LRCs with distance 2 are obtained. Also, constacyclic codes with locality 1 are constructed. They may be so useful in practice thanks to their minimum locality. Constacyclic codes whose locality is equal to their dimension are given. Furthermore, constacyclic LRCs are obtained from cyclotomic cosets.

Accelerating erasure coding by exploiting multiple repair paths in distributed storage systems

Accelerating erasure coding by exploiting multiple repair paths in distributed storage systems

Scalable Distributed Storage Systems: Comparative Study of Key Technologies

Scalable distributed storage systems play a crucial role in modern computing environments, providing efficient and reliable storage solutions for handling vast amounts of data. This paper presents a comprehensive comparative study of key technologies used in scalable distributed storage systems, aiming to provide insights into their strengths, weaknesses, and suitability for different use cases. The paper begins by discussing the importance of scalable distributed storage systems in meeting the growing demands of data-intensive applications, highlighting the need for high performance, fault tolerance, and scalability. It then examines various technologies employed in these systems, including distributed file systems, object storage, and NoSQL databases, analysing their architectures, data models, and scalability mechanisms.

S-MBR: An Efficient Scaling Scheme for PM-MBR-Coded Distributed Storage System

<p>With the rapid increase in data storage, distributed storage systems need to add new nodes to reduce storage and computation pressure. However, the existing scaling schemes are not efficient enough. To solve these issues, A scaling scheme called S-MBR is proposed, which has two notable features. Firstly, S-MBR achieves uniform data distribution by using a symmetric data layout to evenly place data blocks on the expanded data nodes. Secondly, S-MBR minimizes data movement during data redistribution and parity update processes by re-locating data blocks to symmetric positions and performing partial parity block calculations after the storage nodes are expanded. S-MBR can reach the lower bound of data migration volume when the number of nodes required for reconstruction is equal to the number of nodes required for repair. According to mathematical analysis and experimental results, compared to many typical scaling schemes, S-MBR can reduce data transmission volume by up to approximately 93% and average response time by around 41%.</p> <p> </p>

Methodology for Energy Management in a Smart Microgrid Based on the Efficiency of Dispatchable Renewable Generation Sources and Distributed Storage Systems

This paper presents a methodology for energy management in a smart microgrid based on the efficiency of dispatchable generation sources and storage systems, with three different aims: elimination of power peaks; optimisation of the operation and performance of the microgrid; and reduction of energy consumption from the distribution network. The methodology is based on four steps: identification of elements of the microgrid, monitoring of the elements, characterization of the efficiency of the elements, and finally, microgrid energy management. A specific use case is shown at CEDER-CIEMAT (Centro para el Desarrollo de las Energías Renovables—Centro de Investi-gaciones Energéticas, Medioambientales y Tecnológicas), where consumption has been reduced during high tariff periods and power peaks have been eliminated, allowing an annual reduction of more than 25,000 kWh per year, which is equal to savings of more than 8500 €. It also allows the power contracted from the distribution company by CEDER (135 kW) not to be exceeded, which avoids penalties in the electricity bill.

FPGA-Accelerated Erasure Coding Encoding in Ceph Based on an Efficient Layered Strategy

Distributed storage systems such as Ceph have been widely adopted, with erasure coding technology being an essential fault-tolerance technique. While ensuring data reliability and security, it significantly reduces the cost of data storage. Due to the computational overhead and encoding latency introduced by the erasure coding process, the data encoding rate is often constrained. To address this issue, an FPGA-accelerated erasure coding encoding scheme in Ceph, based on an efficient layered strategy (FPGA-Accelerated Erasure Coding Encoding in Ceph with an Efficient Layered Strategy, LFEC-Accelerator), is proposed and implemented. This approach takes full advantage of FPGA’s parallel computing capabilities to accelerate the erasure coding algorithm at the hardware level. Furthermore, to maximize the utilization of the FPGA controller’s resources and ensure that all processing steps are properly managed and scheduled, our approach introduces a hierarchical structure comprising a communication interface layer, task scheduling layer, and hardware acceleration layer. Experimental results indicate that, under the same erasure coding configurations and file sizes, our solution outperforms native Ceph-supported erasure coding libraries such as Jerasure, Clay, Shec and ISA, with an encoding rate improvement of up to 3.04 times.

Innovations in Cyber Security Algorithms for Databases Enhancing Data Retrieval and Management

The term Innovations in Cyber Security Algorithms for Databases Enhancing Data Retrieval and Management refers to a book that studies novel techniques for tackling problems related to digital data. The integration of three complicated methods—DQO, DSS, and RAI—is the major focus of attention in this piece of writing. DQO makes use of machine learning to optimize query processing on the fly to meet fluctuating workloads. This is done to accommodate such workloads. To address issues pertaining to the scale of distributed systems, distributed storage systems (DSS) convey data in an effective manner by utilizing consistent hashing. The RAI algorithm adjusts the index architecture in response to the query patterns to achieve real-time flexibility. In this way, the process of looking for information that is frequently asked about is sped up. The methodology that has been suggested is superior to six different ways that are often used in terms of its adaptability, scalability, and real-time capabilities. This article will give a thorough model for improving data management in computer systems. The objective of this essay is to present the model.