Parallel Database Systems Research Articles

Data Communication Algorithm of HPDB Parallel Database System Based on Computer Network

High-Performance Database (HPDB) is designed to meet the demands of today’s modern data-driven world, as massive volumes of information need to be accessed and analyzed with minimal latency. A pivotal aspect of their operation lies in efficient data communication among nodes in a computer network, which is essential for parallel database systems. HPDBs may involve distributed architectures and parallel database systems to store and process data across multiple nodes or servers in a network. Hence, an algorithm called Hybridized Partitioning Strategy (HPS)-based Communication (C) for achieving HPDB has been proposed to facilitate data transmission and coordination across a computer network using the Message Passing Interface (MPI) protocol. The proposed HPS-C-HPDB technique includes partitioning and distributing data, query routing, and load balancing strategies to achieve high-performance levels. The (HPS) combines hash and range partitioning methods for effective processing and retrieval to balance data distribution, reduce communication overhead in parallel databases, and improve the system’s performance. In parallel database systems, query routing effectively routes requests into the optimal nodes or partitions based on the query’s conditions and the data’s placement and guarantees efficient data processing and retrieval. The proposed scheme is evaluated using various performance metrics like throughput, response time, speedup, and communication overhead analysis.

Retracted: Data Processing Method of Distributed Parallel Database System Based on Wireless Network

Retracted: Data Processing Method of Distributed Parallel Database System Based on Wireless Network

Dynamic Fault Tolerance for Multi-Node Query Processing

Parallel processing is a typical approach to answer analytical queries on large database. As the size of the database increases, we often try to increase the parallelism by incorporating more processing nodes. However, this approach increases the possibility of node failure as well. According to the conventional practice, if a failure occurs during query processing, the database system restarts the query processing from the beginning. Such temporal cost may be unacceptable to the user. This paper proposes a fault-tolerant query processing mechanism, named PhoeniQ, for analytical parallel database systems. PhoeniQ continuously takes a checkpoint for every operator pipeline and replicates the output of each stateful operator among different processing nodes. If a single processing node fails during query processing, another can promptly take over the processing. Hence, PhoneniQ allows the database system to efficiently resume query processing after a partial failure event. This paper presents a key design of PhoeniQ and prototype-based experiments to demonstrate that PhoeniQ imposes negligible performance overhead and efficiently continues query processing in the face of node failure.

Data Processing Method of Distributed Parallel Database System Based on Wireless Network

With the development of society and the arrival of the information age, data processing has become more and more complex, so people need to manage data systems through wireless communication, and distributed systems can effectively improve data analysis, so this paper is based on wireless communication. Distributed database systems are studied. With the rapid development of database systems, how to effectively obtain useful information about massive data has gradually become an important research problem of/with the field of data management. The purpose of this paper is to study how to research the data processing of distributed parallel database system based on wireless network. This paper puts forward the basic concepts of wireless network and distributed parallel database system and proposes a clustering analysis algorithm. The preimproved clustering analysis and the improved distributed parallel clustering analysis are described and compared in detail. From the data in the figure in the experimental part of the text, it can be seen that the efficiency of the database system before the improvement is the lowest at 41% and the highest at 58%. The efficiency of the improved distributed database system is at least 65%, and the highest are 95%. It can be seen that the efficiency of the improved distributed database system is much higher than that of the preimproved database system. So it is very feasible to use the distributed database system to process data.

Design and Implementation of Human-Computer Interaction System in Parallel Digital Library System Based on Neural Network

Information and communication technologies are well thought-out as probable assets for the development of socioeconomics in developing countries. Studies have shown that enhanced infrastructure of telecommunication has facilitated means for underserved population development by various ways. Among the existing applications of ICT, the digital library systems provide with better solutions and respond to a variety of unmet needs of research institutions, scientific communities, and developments. With the development of digital library technology, the parallel database system has become the main tool for efficient information processing in the digital library system. On this basis, based on the parallel environment of the computer cluster, by coordinating the communication in the parallel environment, the coordinator, the collection machine, and the query processor can complete the operation of distribution, load, and maintenance, which has high efficiency and saves much precious time, supports the digital library to meet user requirements effectively, and meets the digital library’s performance requirements for data, and also, the key problem in the parallel algorithm has been solved. The experimental results show that this parallel technique has very good performance and efficiency.

Resource Management Method of Distributed Parallel Database Based on Directed Graph

A distributed parallel database resource management method based on directed graph is proposed. By using content distributor based on distributed unstructured P2P network association, the high performance and stability of distributed parallel database system in dynamic changing environment are guaranteed. The problem of network congestion caused by too many redundant messages and the “barrel effect” caused by resource imbalance due to differential configuration is solved, through resource search algorithm based on directed graph lookahead, and the query node cached the resource information of two-level neighbor nodes. By adopting the Linux cgroups resource management mechanism, fully considering the multi tenant and multi factor based resource scheduling strategy, reduce resource fragments and better meet the problem of distributed parallel database storage or hot spot processing.

Design and Evaluation of an RDMA-aware Data Shuffling Operator for Parallel Database Systems

The commoditization of high-performance networking has sparked research interest in the RDMA capability of this hardware. One-sided RDMA primitives, in particular, have generated substantial excitement due to the ability to directly access remote memory from within an application without involving the TCP/IP stack or the remote CPU. This article considers how to leverage RDMA to improve the analytical performance of parallel database systems. To shuffle data efficiently using RDMA, one needs to consider a complex design space that includes (1) the number of open connections, (2) the contention for the shared network interface, (3) the RDMA transport function, and (4) how much memory should be reserved to exchange data between nodes during query processing. We contribute eight designs that capture salient tradeoffs in this design space as well as an adaptive algorithm to dynamically manage RDMA-registered memory. We comprehensively evaluate how transport-layer decisions impact the query performance of a database system for different generations of InfiniBand. We find that a shuffling operator that uses the RDMA Send/Receive transport function over the Unreliable Datagram transport service can transmit data up to 4× faster than an RDMA-capable MPI implementation in a 16-node cluster. The response time of TPC-H queries improves by as much as 2×.

Similarity query support in big data management systems

Similarity query processing is becoming increasingly important in many applications such as data cleaning, record linkage, Web search, and document analytics. In this paper we study how to provide end-to-end similarity query support natively in a parallel database system. We discuss how to express a similarity predicate in its query language, how to build indexes, how to answer similarity queries (selections and joins) efficiently in the runtime engine, possibly using indexes, and how to optimize similarity queries. One particular challenge is how to incorporate existing similarity join algorithms, which often require a series of steps to achieve a high efficiency, including collecting token frequencies, finding matching record id pairs, and reassembling result records based on id pairs. We present a novel approach that uses existing runtime operators to implement such complex join algorithms without reinventing the wheel; doing so positions the system to automatically benefit from future improvements to those operators. The approach includes a technique to transform a similarity join plan into an efficient operator-based physical plan during query optimization by using a template expressed largely in the system’s user-level query language; this technique greatly simplifies the specification of such a transformation rule. We use Apache AsterixDB, a parallel Big Data management system, to illustrate and validate our techniques. We conduct an experimental study using several large, real datasets on a parallel computing cluster to assess the similarity query support. We also include experiments involving three other parallel systems and report the efficacy and performance results.

A parallel query processing system based on graph-based database partitioning

As parallel database systems have large amounts of data to process, it is important to utilize a scalable and efficient horizontal database partitioning method. The existing partitioning methods have major drawbacks that not only cause large amounts of data redundancy but also still require expensive shuffle operations for join queries in many cases—despite their high data redundancy. We elucidate upon the drawbacks originating from the tree-based partitioning schemes and propose a novel graph-based database partitioning method called GPT that both improves the query performance and reduces data redundancy. We integrate the proposed GPT method into a parallel query processing system, Spark SQL, across all the relevant layers and modules, including the query plan generator and the scan operator. Through extensive experiments using three benchmarks, TPC-DS, IMDB and BioWarehouse, we show that GPT significantly outperforms the state-of-the-art method in terms of both storage overhead and query performance.

Dynamic data reallocation and replication over a cloud environment

SummaryThe performance and efficiency of the distributed database system (DDBS) design are dependent on proper data fragmentation, reallocation, and replication of global relations. In this research, a Cluster Based Distributed and Parallel Database System (CB‐DDBS) architecture over a cloud environment is proposed. The proposed CB‐DDBS architecture processes the client's query requests, which access the clustered DDBS from anywhere. It also allows vertical and horizontal fragmentation, allocation, and replication decisions to be taken statically at the initial stage of the design. In addition, it allows migration and/or replication decisions to be taken by each cluster independently of other clusters using the proposed Optimal Fragment Reallocation and Replication (OFRAR) Algorithm. The proposed CB‐DDBS architecture and the proposed OFRAR Algorithm are tested in both Amazon cloud environment and a simulated environment. Experimental results show that the proposed OFRAR Algorithm efficiently reduces the communication costs for typical access patterns, the overloads of the sites, and the frequency and time spent on fragment migration over the network sites by imposing a stricter condition for fragment reallocation and replication, resulting in a great overall improvement in the DDBS performance. It also shows that the proposed CB‐DDBS architecture results in a significant reduction of the execution time needed for the transactions and for noticing an error.

Big data multi-query optimisation with Apache Flink

Big data analytic frameworks, such as MapReduce, Spark and Flink, have recently gained more popularity to process large data. Flink is an open-source Apache-hosted big data analytic framework for processing batch and streaming data. For historical data processing (batch), Flink's query optimiser is built based on techniques which have been used in the parallel database systems. Flink query optimiser translates the queries into jobs which are repeatedly submitted with similar tasks. Therefore, exploiting the similarity of tasks can avoid redundant computation. In this paper, Flink multi-query optimisation system, Flink-MQO, has been proposed and built on top of Flink software stack. It is considered as an add-on to Apache Flink to optimise multi-query based on data sharing. The Flink-MQO system exploits the data sharing opportunities of selection operators to eliminate the redundancy and duplication of data in-network movement of multi-query. Experimental results show that the exploiting of shared selection operators in big data multi-query can provide promising query execution time. Therefore, Flink-MQO system can potentially be used in the stream processing to improve the performance of the real-time applications.

Scalable parallel graph algorithms with matrix–vector multiplication evaluated with queries

Graph problems are significantly harder to solve with large graphs residing on disk compared to main memory only. In this work, we study how to solve four important graph problems: reachability from a source vertex, single source shortest path, weakly connected components, and PageRank. It is well known that the aforementioned algorithms can be expressed as an iteration of matrix–vector multiplications under different semi-rings. Based on this mathematical foundation, we show how to express the computation with standard relational queries and then we study how to efficiently evaluate them in parallel in a shared-nothing architecture. We identify a common algorithmic pattern that unifies the four graph algorithms, considering a common mathematical foundation based on sparse matrix–vector multiplication. The net gain is that our SQL-based approach enables solving “big data” graph problems on parallel database systems, debunking common wisdom that they are cumbersome and slow. Using large social networks and hyper-link real data sets, we present performance comparisons between a columnar DBMS, an open-source array DBMS, and Spark’s GraphX.

Parallel Database for Student Counselling through Single Window System for Admission in Engineering Colleges

Nowadays, the growth of the educational sector generates massive data such as details of students, staff and their academic performance, administrative and research details, etc. These data are stored in distributed databases. The parallel database performs parallel query processing hence, the query processing time is reduced and throughput for the transaction processing is improved. The parallel database system also improves the performance by employing parallelism through various database management operations such as loading the data, building indexes, and evaluating queries. The data is partitioned and placed across multiple disks for parallel input and output (I/O) operation to achieve parallelism on queries using pipeline parallelism. The individual relational operations such as sort, join, aggregation, etc. are executed in parallel database by each processor that works independently with its own partition. Queries are expressed in high level language such as structured query language (SQL) that are translated into relational algebra for query processing. Thus, the parallel database increases the throughput in database management and reduces the query processing time. This paper presents a parallel database for student counselling for engineering university admission using MySQL relational database management system.

Horizontal partitioning method for test verification in parallel database systems

In parallel database systems the partitioning methods considered in current researches are static. This research paper presents a partitioning method to divide the database relations into dynamic horizontal partitions. Every partition contains some tuples of the database relation. These partitions will be checked using the subtests generated from the integrity constraints during the test verification process. Furthermore, the integrity of these partitions will be enforced by the generated integrity rules. It presents the algorithm of distributing the integrity test(s) among the horizontal partitions. As our intention is to parallelise the execution of the integrity tests, the relations specified in the test are dynamically partitioned into several parts based on the number of available processors. Each processor checks the validity of the test concurrently by accessing the partition assigned to it.

Horizontal partitioning method for test verification in parallel database systems

In parallel database systems the partitioning methods considered in current researches are static. This research paper presents a partitioning method to divide the database relations into dynamic horizontal partitions. Every partition contains some tuples of the database relation. These partitions will be checked using the subtests generated from the integrity constraints during the test verification process. Furthermore, the integrity of these partitions will be enforced by the generated integrity rules. It presents the algorithm of distributing the integrity tests among the horizontal partitions. As our intention is to parallelise the execution of the integrity tests, the relations specified in the test are dynamically partitioned into several parts based on the number of available processors. Each processor checks the validity of the test concurrently by accessing the partition assigned to it.

Resource bricolage and resource selection for parallel database systems

Running parallel database systems in an environment with heterogeneous resources has become increasingly common, due to cluster evolution and increasing interest in moving applications into public clouds. Performance differences among machines in the same cluster pose new challenges for parallel database systems. First, for database systems running in a heterogeneous cluster, the default uniform data partitioning strategy may overload some of the slow machines, while at the same time it may underutilize the more powerful machines. Since the processing time of a parallel query is determined by the slowest machine, such an allocation strategy may result in a significant query performance degradation. Second, since machines might have varying resources or performance, different choices of machines may lead to different costs or performance for executing the same workload. By carefully selecting the most suitable machines for running a workload, we may achieve better performance with the same budget, or we may meet the same performance requirements with a lower cost. We address these challenges by introducing techniques we call resource bricolage and resource selection that improve database performance in heterogeneous environments. Our approaches quantify the performance differences among machines with various resources as they process workloads with diverse resource requirements. For the purpose of better resource utilization, we formalize the problem of minimizing workload execution time and view it as an optimization problem, and then, we employ linear programming to obtain a recommended data partitioning scheme. For the purpose of better resource selection, we formalize two problems: One minimizes the total workload execution time with a given budget, and the other minimizes the total budget with a given performance target. We then employ different mixed-integer programs to search for the optimal resource selection decisions. We verify the effectiveness of both resource bricolage and resource selection techniques with an extensive experimental study.

Resource Bricolage for Parallel DBMSs on Heterogeneous Clusters

Running parallel database systems in an environment with heterogeneous resources has become increasingly common, due to cluster evolution and increasing interest in moving applications into public clouds or shared infrastructures. For database systems running in a heterogeneous cluster, the default uniform data partitioning strategy may overload some of the slow machines while at the same time it may underutilize the more powerful machines. Since the processing time of a parallel query is determined by the slowest machine, such an allocation strategy may result in a significant query performance degradation. We take a first step to address this problem by introducing a technique we call resource bricolage that improves database performance in heterogeneous environments. Our approach quantifies the performance differences among machines with various resources as they process workloads with diverse resource requirements. We formalize the problem of minimizing workload execution time and view it as an optimization problem, and then we employ linear programming to obtain a recommended data partitioning scheme. We verify the effectiveness of our technique with an extensive experimental study on a commercial database system.

Применение графических ускорителей для обработки запросов над сжатыми данными в параллельных системах баз данных

This article talks about using graphics processors for query processing in parallel database systems. The goal is to evaluate query execution efficiency over compressed database without decompression on multicore GPUs which support CUDA technology. GPU’s memory size is significantly smaller than modern computer system’s RAM size. This fact affects database’s size can be loaded into GPU’s internal memory, thus computing potential of GPU can not be used efficiently. The new approach presented in this article allows query processing over compressed data on GPU. An emulator of parallel DBMS is developed based on this approach. The similar emulator for a CPU is designed. Results of computational experiments are presented and analysis of efficiency of the proposed approaches is performed.

Алгоритм репрезентативного сэмплинга для систем баз данных на основе фрагментного параллелизма

Sampling is a popular approach to very large databases processing in a wide range of applications, e.g. data mining, histograms construction, query execution cost estimation, etc. Use of either the sample instead of the original database can reduce the accuracy of the results, but offset by a reduction of time executing processing. Representative sampling allows you to save the sample of certain characteristics of the database. However, existing algorithms for representative sampling can not be used for pas-parallel database systems because it does not take into account the characteristics of the data distribution fissionable by the compute nodes of the cluster system. In this paper we propose al-representative sampling algorithm for parallel relational database systems based on the slice of parallelism. The results of computational experiments on the proposed algorithm, showing adequate maintenance of representativity database properties distributed across the nodes of a cluster system.

Resource bricolage for parallel database systems

Running parallel database systems in an environment with heterogeneous resources has become increasingly common, due to cluster evolution and increasing interest in moving applications into public clouds. For database systems running in a heterogeneous cluster, the default uniform data partitioning strategy may overload some of the slow machines while at the same time it may under-utilize the more powerful machines. Since the processing time of a parallel query is determined by the slowest machine, such an allocation strategy may result in a significant query performance degradation. We take a first step to address this problem by introducing a technique we call resource bricolage that improves database performance in heterogeneous environments. Our approach quantifies the performance differences among machines with various resources as they process workloads with diverse resource requirements. We formalize the problem of minimizing workload execution time and view it as an optimization problem, and then we employ linear programming to obtain a recommended data partitioning scheme. We verify the effectiveness of our technique with an extensive experimental study on a commercial database system.