Fault-tolerant Data Storage Research Articles

Data Warehouse Failover Cluster for Analytical Queries in Banking

Introduction. The banking sector assigns high priority to data storage, as it is a critical aspect of business operations. The volume of data in this area is steadily growing. With the increasing volume of data that needs to be stored, processed and analyzed, it is critically important to select a suitable data storage solution and develop the required architecture. The presented research is aimed at filling the gap in the existing knowledge of the data base management system (DBMS) suitable for the banking sector, as well as to suggest ways for a fault-tolerant data storage cluster. The purpose of the work is to analyze the key DBMS for analytical queries, determine the priorities of the DBMS for the banking sector, and develop a fault-tolerant data storage cluster. To meet the performance and scalability requirements, a data storage solution with a fault-tolerant architecture that meets the requirements of the banking sector has been proposed.Materials and Methods. Domain analysis allowed us to create a set of characteristics that a DBMS for analytical queries (OnLine Analytical processing — OLAP) should correspond to, compare some popular DBMS OLAP, and offer a fault-tolerant cluster configuration written in xml, supported by the ClickHouse DBMS. Automation was done using Ansible Playbook. It was integrated with the Gitlab version control system and Jinja templates. Thus, rapid deployment of the configuration on all nodes of the cluster was achieved.Results. For OLAP databases, criteria were developed and several popular systems were compared. As a result, a reliable cluster configuration that met the requirements of analytical queries has been proposed for the banking industry. To increase the reliability and scalability of the DBMS, the deployment process was automated. Detailed diagrams of the cluster configuration were also provided.Discussion and Conclusions. The compiled criteria for the DBMS OLAP allowed us to determine the need for this solution in the organization. Comparison of popular DBMS can be used by organizations to minimize costs when selecting a solution. The proposed configuration of the data warehouse cluster for analytical queries in the banking sector will improve the reliability of the DBMS and meet the requirements for subsequent scalability. Automation of cluster deployment by the mechanism of templating configuration files in Ansible Playbook provides configuring a ready-made cluster on new servers in minutes.

PyRS: Cross-platform Data Fault-tolerant Storage Library Based on RS Erasure Code

<p>Erasure code has been used by more and more researchers to solve the problem of efficient, reliable, and fault-tolerant data storage. However, the existing libraries based on erase code can only run on the Linux platform, and some of them need GPU support. This paper implements a cross-platform data fault-tolerant storage library based on RS erasure code, PyRS, which is running on CPU without GPU support and developed in Python. PyRS uses Vandermonde matrix as the coding matrix and Numba and NumPy libraries to speed up and optimize the program. This paper compares PyRS with Jerasure on the same Linux platform. The results show that the encoding and decoding speed of PyRS is 8 times faster than that of Jerasure. The CPU usage rate of both is about 25% and the memory usage rate of PyRS is about 5% higher than that of Jerasure. The same experiments are carried out on PyRS on the Windows platform. Experimental results show that compared with running on the Linux platform, PyRS running on the Windows platform has almost the same speed of encoding and decoding, and its CPU usage rate increases by about 15%, while its memory usage rate decreases by about 5%.</p> <p> </p>

TH-DPMS

The rapidly increasing data in recent years requires the datacenter infrastructure to store and process data with extremely high throughput and low latency. Fortunately, persistent memory (PM) and RDMA technologies bring new opportunities towards this goal. Both of them are capable of delivering more than 10 GB/s of bandwidth and sub-microsecond latency. However, our past experiences and recent studies show that it is non-trivial to build an efficient and distributed storage system with such new hardware. In this article, we design and implement TH-DPMS (<underline>T</underline>sing<underline>H</underline>ua <underline>D</underline>istributed <underline>P</underline>ersistent <underline>M</underline>emory <underline>S</underline>ystem) based on persistent memory and RDMA, which unifies the memory, file system, and key-value interface in a single system. TH-DPMS is designed based on a unified distributed persistent memory abstract, pDSM. pDSM acts as a generic layer to connect the PMs of different storage nodes via high-speed RDMA network and organizes them into a global shared address space. It provides the fundamental functionalities, including global address management, space management, fault tolerance, and crash consistency guarantees. Applications are enabled to access pDSM with a group of flexible and easy-to-use APIs by using either raw read/write interfaces or the transactional ones with ACID guarantees. Based on pDSM, we implement a distributed file system and a key-value store named pDFS and pDKVS, respectively. Together, they uphold TH-DPMS with high-performance, low-latency, and fault-tolerant data storage. We evaluate TH-DPMS with both micro-benchmarks and real-world memory-intensive workloads. Experimental results show that TH-DPMS is capable of delivering an aggregated bandwidth of 120 GB/s with 6 nodes. When processing memory-intensive workloads such as YCSB and Graph500, TH-DPMS improves the performance by one order of magnitude compared to existing systems and keeps consistent high efficiency when the workload size grows to multiple terabytes.

MATHEMATICAL MODELS OF CREATION OF A SUBSYSTEM OF ENSURING SAFETY OF INFORMATION IN THE DISTRIBUTED INFORMATION SYSTEMS

Practical implementation of the concept of a distributed information system requires the implementation of a complex scientific-technical problems related to the choice of composition and use of storage and processing of information, creation and use of common data arrays, information security, etc. Fundamentally, a modern distributed information system is a system of communication and information transmission as well as a storage and data processing system. Storage and data processing system is designed for organization of reliable and fault-tolerant data storage, high-performance servers to access storage devices and information processing. The measure of quality, and in some cases indicator of functioning of data storage and processing system is proposed. The tasks that need to be tackled when constructing and operating the storage and processing requirements of highly reliable data storage, as well as their following configurations: determination of the number and location of centers of data storage and processing in distributed information systems; choice of the composition of the complex used for storage in the storage centers and data processing, with the organization of a backup subsystem, the backup and restore data. The task of storage structure optimization according to the criterion of maximum values of the readiness factor of the storage system is formed. This problem is reduced to the kind of problems of integer linear programming with Boolean variables, this fact allows to apply the existing methods for its solvation. A method for determining the rational level of expenditure on the formation of the complex used for storage in the storage and data processing system based on the use of elements of probability theory and the theory of well-being (the principle of Pareto optimality) is developed. The optimal solution of the problem by optimizing a linear convolution is obtained.

Coding for Reliable Data Storage on Different Hardware Platforms

When systems are designed to tolerate faulty components, application data must be protected against loss. This is reached by a distribution of data together with addition of redundant elements according to an erasure-tolerant code. In this paper, we elaborate architectures for such a fault-tolerant data storage. The concepts are originated from distributed systems and mostly implemented by software. We extend these concepts for usage in the scope of system on chip architectures. On the one hand, systems on chips, and multi core systems are employed as a platform for code calculation - on the other hand, such architectures include these techniques to fulfill their own functionality. We explain how data coding is mapped to (i) multi core CPU structures and (ii) implemented in a specialized design on a FPGA. We compare the time for coding on these architectures for a Cauchy-Reed/Solomon and a classical Reed/Solomon code.

Transversals of Vertex Partitions in Graphs

This paper studies graph properties of the following forms: For every partition of the vertex set that satisfies an upper (or lower) bound on the number of elements in each partition class, there is a transversal of the partition that is an independent (or dominating) set. A possible application to fault-tolerant data storage is discussed, and bounds for the parameters that are functions of minimum and maximum degree are established. The complexity of associated decision problems is also addressed.