Parallel Query Research Articles

ExaHDF5: Delivering Efficient Parallel I/O on Exascale Computing Systems

Scientific applications at exascale generate and analyze massive amounts of data. A critical requirement of these applications is the capability to access and manage this data efficiently on exascale systems. Parallel I/O, the key technology enables moving data between compute nodes and storage, faces monumental challenges from new applications, memory, and storage architectures considered in the designs of exascale systems. As the storage hierarchy is expanding to include node-local persistent memory, burst buffers, etc., as well as disk-based storage, data movement among these layers must be efficient. Parallel I/O libraries of the future should be capable of handling file sizes of many terabytes and beyond. In this paper, we describe new capabilities we have developed in Hierarchical Data Format version 5 (HDF5), the most popular parallel I/O library for scientific applications. HDF5 is one of the most used libraries at the leadership computing facilities for performing parallel I/O on existing HPC systems. The state-of-the-art features we describe include: Virtual Object Layer (VOL), Data Elevator, asynchronous I/O, full-featured single-writer and multiple-reader (Full SWMR), and parallel querying. In this paper, we introduce these features, their implementations, and the performance and feature benefits to applications and other libraries.

OLAP parallel query processing in clouds with C‐ParGRES

SummaryThe advent of big data technologies has changed the way many companies manage their data. Several companies moved their data to the cloud using the concept of database‐as‐a‐service (DBaaS). Moving databases to the cloud presents several challenges related to flexible and scalable management of data. Although some of these companies migrated to NoSQL databases, most still rely on relational databases in the cloud to manage data, especially data that is critical to the decision making process. Online analytical processing (OLAP) queries take a long time to be processed, thus demanding high‐performance capabilities from their associated database systems to get results in a feasible time. In this article, we propose a middleware solution that can be deployed in any cloud provider, named C‐ParGRES, which explores database replication and interquery and intraquery parallelism to efficiently support OLAP queries in the cloud. C‐ParGRES is an extension of ParGRES, an open‐source database cluster middleware for high‐performance OLAP query processing in clusters. C‐ParGRES exploits cloud capabilities such as on‐demand resource provisioning and elasticity. In addition, C‐ParGRES can create multiple and independent virtual clusters for different database and users. We evaluate C‐ParGRES with two real‐world OLAP applications, both from the Brazilian Institute of Geography and Statistics. Results show that C‐ParGRES is a cost‐effective solution for OLAP query processing in the cloud.

Research on partitioning algorithm based on RDF graph

SummaryWith the increasing size of the RDF dataset, the application of parallel computing frameworks is applying more and more widely in big data. In order to realize the parallel query of data, an edge partitioning algorithm is proposed based on RDF sentence graph. Firstly, the algorithm transforms the RDF graph into an RDF sentence graph using coarsening and partitioning. Secondly, the minimum degree vertex partitioning algorithm is put forward to partition the RDF sentence graph. Finally, according to the equivalence thought between edge segmentation of RDF sentence and vertex segmentation of RDF graph, the intersection between RDF subgraphs is the vertex cut set of RDF graph to achieve parallel query of RDF data. The experimental results show that the algorithm's partitioning time and efficiency are better than the traditional algorithms.

An analytic study of architecture, security, privacy, query processing, and performance evaluation of database‐as‐a‐service

AbstractIn the present era of technology, the pace of data generation is very high, and the biggest challenge is the manageability of data with efficient and secure storage options as well as at lower cost. Therefore, in this article, we have elaborated the database‐as‐a‐service (DaaS) concept having its introduction, architecture, security and privacy, query processing, and an analytic performance evaluation relating to query performance on encrypted data of DaaS. We have conducted an analytic study of the state‐of‐the‐art by analyzing CPU and query execution time. According to this evaluation, it is concluded that data encryption and decryption need heavy processing power and are time‐consuming. Moreover, there is several query execution problems that exist on encrypted data. A major improvement in query performance may be achieved through parallel query execution after the deployment of multithreading having maximum possible threads.

Lock-free Data Structures for Data Stream Processing

Processing data in real-time instead of storing and reading from tables has led to a specialization of DBMS into the so-called data stream processing paradigm. While high throughput and low latency are key requirements to keep up with varying stream behavior and to allow fast reaction to incoming events, there are many possibilities how to achieve them. In combination with modern hardware, like server CPUs with tens of cores, the parallelization of stream queries for multithreading and vectorization is a common schema. High degrees of parallelism, however, need efficient synchronization mechanisms to allow good scaling with threads for shared memory access.In this work, we identify the most time-consuming operations for stream processing exemplarily for our own stream processing engine PipeFabric. In addition, we present different design principles of lock-free data structures which are suited to overcome those bottlenecks. We will finally demonstrate how lock-freedom greatly improves performance for join processing and tuple exchange between operators under different workloads. Nevertheless, the efficient usage of lock-free data structures comes with additional efforts and pitfalls, which we also discuss in this paper.

PathQuery Pregel: high-performance graph query with bulk synchronous processing

High-performance graph query systems are a scalable way to mine information in Knowledge Graphs, especially when the queries benefit from a high-level expressive query language. This paper presents techniques to algorithmically compile queries expressed in a high-level language (e.g., Datalog) into a directed acyclic graph query plan and details how these queries can be run on a Pregel graph vertex-centric compute system. Our solution, called PathQuery Pregel, creates plans for any conjunctive or disjunctive queries with aggregation and negation; we describe how the query execution extracts graph results optimally while avoiding many join operations where parallel map execution is permitted. We provide details of how we scaled this system out to execute large set of queries in parallel over the Google Knowledge Graph, a graph of 70B edges, or facts; we describe our production experience with PathQuery Pregel.

BlackMagic

Relational DBMSs allow users to extend the standard declarative SQL language surface using User Defined Functions (UDFs) that implement custom behavior. While UDFs offer many advantages, it is well-known amongst practitioners that they can cause severe degradation in query performance. This degradation is due to the fact that state-of-the-art query optimizers treat UDFs as black boxes and do not reason about them during optimization. We demonstrate Froid, a framework for optimizing UDFs by opening up this black box and exposing its underlying operations to the query optimizer. It achieves this by systematically translating the entire body of an imperative multi-statement UDF into a single relational algebraic expression. Thereby, any query invoking this UDF is transformed into a query with a nested sub-query that is semantically equivalent to the UDF. We then leverage existing sub-query optimization techniques and thereby get efficient, set-oriented, parallel query plans as opposed to inefficient, iterative, serial execution of UDFs. We demonstrate the benefits of Froid including performance gains of up to multiple orders of magnitude on real workloads. Froid is available as a feature of Microsoft SQL Server 2019 called 'Scalar UDF Inlining'.

An improved algorithm for clustering uncertain traffic data streams based on Hadoop platform

During the development of intelligent transportation systems, traffic data has the characteristics of streaming, high dimension and uncertainty. In order to realize the query of uncertain traffic data streams in a distributed environment, the authors design the algorithm of Uncertain Traffic Data Stream Parallel Continuous Query algorithm (UTDSPCQ). Firstly, the sliding window mode is applied to realize the data receiving and buffering in the data stream environment, so as to adapt to the MapReduce computing framework of the Hadoop distributed structure. Then, the impact of the high dimensionality and uncertainty of the data on the feature analysis of the dataset is reduced, through the dimension reduction and data rewriting. Finally, a multi-attribute data point RePoint is newly defined, to solve the problem of data dimension increase caused by data rewriting. Experiments show that this algorithm optimizes the traditional density-based clustering algorithm, and make it more adaptable to parallel continuous queries for uncertain traffic data streams, and can fully consider the newly generated streaming traffic data.

Massively Distributed Time Series Indexing and Querying

Indexing is crucial for many data mining tasks that rely on efficient and effective similarity query processing. Consequently, indexing large volumes of time series, along with high performance similarity query processing, have became topics of high interest. For many applications across diverse domains though, the amount of data to be processed might be intractable for a single machine, making existing centralized indexing solutions inefficient. We propose a parallel indexing solution that gracefully scales to billions of time series, and a parallel query processing strategy that, given a batch of queries, efficiently exploits the index. Our experiments, on both synthetic and real world data, illustrate that our index creation algorithm works on four billion time series in less than five hours, while the state of the art centralized algorithms do not scale and have their limit on 1 billion time series, where they need more than five days. Also, our distributed querying algorithm is able to efficiently process millions of queries over collections of billions of time series, thanks to an effective load balancing mechanism.

ALGORITHMS FOR MANAGING THE LOGICAL STRUCTURE OF THE DATABASE USING A PARAMETRIC MODEL OF COMPETITIVE ACCESS TO QUERIES BASED ON THE RANDOM FOREST METHOD

In article discusses the approach to development of mathematical software for support the process of managing the data schema in relational DBMS in terms of processing of parallel queries stream that compete for data in the hierarchy of the DBMS core memory. The necessity of the formation of a parametric model of queries competitive access. Briefly discusses methods of machine learning, allowing to solve the problem of regression recovery. The use of the random forest method as the most universal method of approximation of arbitrary functions is substantiated. A method of forming a parametric model of competitive access based on the random forest method, as well as an approach with the ensemble of sets of decision trees, which allows to provide the required generalizing ability and stability of the model to partial features and diversity of all types of queries received at the input of the DBMS. The stages of the developed algorithms are presented: ranking query parameters by total execution time and automatic data distribution, allowing you to go from approximating the target system with linear-continuous functions to a set of logical data schema objects, ordered by their effect on time, total query execution time, reducing multi-criteria optimization task to a task optimization by one criterion.

Parallel Querying of Distributed Ontologies with Shared Vocabulary

Parallel Querying of Distributed Ontologies with Shared Vocabulary

Parallel query execution over encrypted data in database-as-a-service (DaaS)

The main challenge in database-as-a-service is the security and privacy of data because service providers are not usually considered as trustworthy. So, the data must be encrypted before storing into the database. Another challenge arises that the performance is degraded on the deployment of encryption algorithm on runtime. Furthermore, the connectivity through the Internet adds more delay. To tackle this, we have proposed parallel query execution methodology using multithreading technique up to 6 threads. We have conducted experiments up to 1000,000 (1 million) encrypted records. Our results are quite promising. For data encryption/decryption, we have used advance encryption standard with blocking length of 256 bits. We have designed our methodology in the context of parallel computation method proposed in the literature (Ho et al., in: Proceedings of the 2017 international conference on machine learning and soft computing, pp 47–52, 2017). We compared the results with state-of-art algorithms. The state-of-art algorithms execute the experiments on 10,120 encrypted records maximum which took about time of 1000 ms with 2 threads. But the proposed methodology is proved outstanding that executed the experiments which were performed on 100,000 encrypted records. It outperformed with 6 threads which took only 507 ms even with 2 threads, and the proposed methodology is much better which took only 994 ms. So, the efficiency and scalability of the proposed methodology are proved better as compared to state-of-the-art algorithms.

DFTHR: A Distributed Framework for Trajectory Similarity Query Based on HBase and Redis

In recent years positioning sensors have become ubiquitous, and there has been tremendous growth in the amount of trajectory data. It is a huge challenge to efficiently store and query massive trajectory data. Among the typical operation over trajectories, similarity query is an important yet complicated operator. It is useful in navigation systems, transportation optimizations, and so on. However, most existing studies have focused on handling the problem on a centralized system, while with a single machine it is difficult to satisfy the storage and processing requirements of mass data. A distributed framework for the similarity query of massive trajectory data is urgently needed. In this research, we propose DFTHR (distributed framework based on HBase and Redis) to support the similarity query using Hausdorff distance. DFTHR utilizes a segment-based data model with a number of optimizations for storing, indexing and pruning to ensure efficient querying capability. Furthermore, it adopts a bulk-based method to alleviate the cost for adjusting partitions, so that the incremental dataset can be efficiently supported. Additionally, DFTHR introduces a co-location-based distributed strategy and a node-locality-based parallel query algorithm to reduce the inter-worker cost overhead. Experiments show that DFTHR significantly outperforms other schemes.

THBase: A Coprocessor-Based Scheme for Big Trajectory Data Management

The rapid development of distributed technology has made it possible to store and query massive trajectory data. As a result, a variety of schemes for big trajectory data management have been proposed. However, the factor of data transmission is not considered in most of these, resulting in a certain impact on query efficiency. In view of that, we present THBase, a coprocessor-based scheme for big trajectory data management in HBase. THBase introduces a segment-based data model and a moving-object-based partition model to solve massive trajectory data storage, and exploits a hybrid local secondary index structure based on Observer coprocessor to accelerate spatiotemporal queries. Furthermore, it adopts certain maintenance strategies to ensure the colocation of relevant data. Based on these, THBase designs node-locality-based parallel query algorithms by Endpoint coprocessor to reduce the overhead caused by data transmission, thus ensuring efficient query performance. Experiments on datasets of ship trajectory show that our schemes can significantly outperform other schemes.

Feasibility of Chatbot for Mehran UET Examination Department

A chatbot system enables communication between computer and users through natural language. The university students face many problems for the queries and details regarding their examination and it is also difficult for them to physically visit the examination department frequently. The concerned authorities find it challenging to handle thousands of student queries in parallel, thus, consuming a lot of time. At present, majority of the chatbot system are available in English language. Therefore, people who are not fluent in English language cannot use these systems due to the obvious reason of language barrier. This paper focuses on the viability of a chatbot system for the examination department of Mehran University of Engineering Technology (MUET) Jamshoro. The chatbot discussed in this paper has been designed to support both Urdu and English languages. Three chatbot interfaces have been developed, Urdu typing based chatbot, English typing based chatbot and English voice based Chabot. The typing based chatbots (both English and Urdu) have been developed using AIML, and voice based chatbot is based on dialogflow called as API.ai. This study is not only concerned with the development of three interfaces of chatbot systems, but also covers their evaluation. To find best chatbot interface, a survey was conducted with engineering students of MUET Jamshoro. The results revealed that the among the three chatbot interfaces, English typing-based interface is the best due to the usability, easiness and the general understanding of queries.

Approaches to Speed up Data Processing in Relational Databases

The increasing of data volumes and the tightening of requirements by the time of data processing actualize the problem of finding methods for optimizing data structures and queries in databases. This article presents a set of methods that can help speed up the processing of data. These include the method of restructuring a completed database, a method for obtaining a parallel query plan, and methods for optimizing queries. The presented methods can be used both in a complex and independently of each other. At the end of the article, are shown the results of the experiments, which were carried out on a test database for approbation of optimization methods for a parallel query plan.

Application of parallel algorithm optimisation method to relational queries by reducing interprocessor data exchange time

The article presents the results of studies on the adaptation of methods for optimising parallel algorithms by time and volume of computational resources to queries in relational databases. Research focuses on a method that allows you to improve the execution schedule of a parallel query by execution time due to the redistribution of operations between processes. It leads to a reduction in amount of messages transferred between processors, and to the time spent on transfer of data. It does not affect the amount of computational resources. The method is based on adjacency lists that correspond to the information graph of the algorithm. It can be applied in conjunction with other scheduling methods, for example, focused on computation node count optimisation, to achieve optimal result within multiple parameters.

A Parallel Processing Method for Moving Top-K Spatial Keyword Query

We propose an influential set based moving k keyword query processing model, which avoids the shortcoming of safe region-based approaches that the update cost and update frequency cannot be optimized simultaneously. Based on the model, we design a parallel query processing method and a parallel validation method for multicore processing platforms. The time complexity of the algorithms is O((log|D|+p.k)/p.k) and O(log p.k), respectively, which are all O(1/k) times the time complexity of the state-of-the-art method. The experiment result confirms the superiority of our algorithms over the state-of-the-art method.

Application of parallel algorithm optimisation method to relational queries by reducing interprocessor data exchange time

The article presents the results of studies on the adaptation of methods for optimising parallel algorithms by time and volume of computational resources to queries in relational databases. Research focuses on a method that allows you to improve the execution schedule of a parallel query by execution time due to the redistribution of operations between processes. It leads to a reduction in amount of messages transferred between processors, and to the time spent on transfer of data. It does not affect the amount of computational resources. The method is based on adjacency lists that correspond to the information graph of the algorithm. It can be applied in conjunction with other scheduling methods, for example, focused on computation node count optimisation, to achieve optimal result within multiple parameters.

HetExchange

Modern server hardware is increasingly heterogeneous as hardware accelerators, such as GPUs, are used together with multicore CPUs to meet the computational demands of modern data analytics work-loads. Unfortunately, query parallelization techniques used by analytical database engines are designed for homogeneous multicore servers, where query plans are parallelized across CPUs to process data stored in cache coherent shared memory. Thus, these techniques are unable to fully exploit available heterogeneous hardware, where one needs to exploit task-parallelism of CPUs and data-parallelism of GPUs for processing data stored in a deep, non-cache-coherent memory hierarchy with widely varying access latencies and bandwidth. In this paper, we introduce HetExchange-a parallel query execution framework that encapsulates the heterogeneous parallelism of modern multi-CPU-multi-GPU servers and enables the parallelization of (pre-)existing sequential relational operators. In contrast to the interpreted nature of traditional Exchange, HetExchange is designed to be used in conjunction with JIT compiled engines in order to allow a tight integration with the proposed operators and generation of efficient code for heterogeneous hardware. We validate the applicability and efficiency of our design by building a prototype that can operate over both CPUs and GPUs, and enables its operators to be parallelism- and data-location-agnostic. In doing so, we show that efficiently exploiting CPU-GPU parallelism can provide 2.8x and 6.4x improvement in performance compared to state-of-the-art CPU-based and GPU-based DBMS.