Query Plan Research Articles

Improved continuous query plan with cluster weighted dominant querying in synthetic datasets

The arrival of large voluminous continuous queries sets for a given query leads an insignificant insights. The elimination of certain data tuples occurs in order to balance the system load. The streaming query removes the improper data tuples and uses proper data tuples in the form of defined tables or sets. However, major drawback occurs due to unbounded streaming and inadequate access to end data. Due to such constraints, many stream processing methods makes the processed data unavailable for any applications or to the related queries of neighborhood branches. This paper avoids such problems during the process of data tuples at the generation of queries. The study uses a streaming model that executes effective query plans in continuous data. The streaming model aims reduce the communication cost and improves the scalability of continuous aggregation queries. It sub-divides the client query and executes it over data aggregators within the incoherent limit. A weighted dominant query algorithm is formulated to provide the top dominant value in each sub-query clusters. This reduces the cost for computation in synthetic databases. The experimental results proved that the proposed model with weighted dominant query algorithm effectively improves scalability by reducing the computational cost.

Stylus

RDF is one of the most commonly used knowledge representation forms. Many highly influential knowledge bases, such as Freebase and PubChemRDF, are in RDF format. An RDF data set is usually represented as a collection of subject-predicate-object triples. Despite the flexibility of RDF triples, it is challenging to serve SPARQL queries on RDF data efficiently by directly managing triples due to the following two reasons. First, heavy joins on a large number of triples are needed for query processing, resulting in a large number of data scans and large redundant intermediate results; Second, weakly-typed triple representation provides suboptimal random access - typically with logarithmic complexity. This data access challenge, unfortunately, cannot be easily met by a better query optimizer as large graph processing is extremely I/O-intensive. In this paper, we argue that strongly-typed graph representation is the key to high-performance RDF query processing. We propose Stylus - a strongly-typed store for serving massive RDF data. Stylus exploits a strongly-typed storage scheme to boost the performance of RDF query processing. The storage scheme is essentially a materialized join view on entities, it thus can eliminate a large number of unnecessary joins on triples. Moreover, it is equipped with a compact representation for intermediate results and an efficient graph-decomposition based query planner. Experimental results on both synthetic and real-life RDF data sets confirm that the proposed approach can dramatically boost the performance of SPARQL query processing.

Structural XML Query Processing

Since the boom in new proposals on techniques for efficient querying of XML data is now over and the research world has shifted its attention toward new types of data formats, we believe that it is crucial to review what has been done in the area to help users choose an appropriate strategy and scientists exploit the contributions in new areas of data processing. The aim of this work is to provide a comprehensive study of the state-of-the-art of approaches for the structural querying of XML data. In particular, we start with a description of labeling schemas to capture the structure of the data and the respective storage strategies. Then we deal with the key part of every XML query processing: a twig query join, XML query algebras, optimizations of query plans, and selectivity estimation of XML queries. To the best of our knowledge, this is the first work that provides such a detailed description of XML query processing techniques that are related to structural aspects and that contains information about their theoretical and practical features as well as about their mutual compatibility and general usability.

Multi-objective parametric query optimization

We propose a generalization of the classical database query optimization problem: multi-objective parametric query (MPQ) optimization. MPQ compares alternative processing plans according to multiple execution cost metrics. It also models missing pieces of information on which plan costs depend upon as parameters. Both features are crucial to model query processing on modern data processing platforms. MPQ generalizes previously proposed query optimization variants, such as multi-objective query optimization, parametric query optimization, and traditional query optimization. We show, however, that the MPQ problem has different properties than prior variants and solving it requires novel methods. We present an algorithm that solves the MPQ problem and finds, for a given query, the set of all relevant query plans. This set contains all plans that realize optimal execution cost tradeoffs for any combination of parameter values. Our algorithm is based on dynamic programming and recursively constructs relevant query plans by combining relevant plans for query parts. We assume that all plan execution cost functions are piecewise-linear in the parameters. We use linear programming to compare alternative plans and to identify plans that are not relevant. We present a complexity analysis of our algorithm and experimentally evaluate its performance.

The Merkurion approach for similarity searching optimization in Database Management Systems

Modern Database Management Systems (DBMSs) retrieve songs that resemble those in a music dataset, identify plagiarism in a set of documents, or provide past cases to physicians by taking into account the characteristics of a query exam. All such tasks require the comparison of data by similarity, which can be expressed in terms of distance-based queries in metric spaces. Traditional query processing relies mostly on histograms for describing the data distribution space and choosing a data retrieval path that quickly leads to the answer, discarding comparisons of most unwanted data. However, DBMSs still lack adequate support for selectivity estimation of query operators for data types embedded in metric spaces. This article addresses a novel strategy that extends the query optimizer of a DBMS, so that it can also perform both logical and physical query plan optimizations in searches that include similarity predicates. The proposal, named Merkurion, updates the concept of Data Distribution Space and captures data distributions according to the distances between the elements within a dataset. Moreover, it employs concise representations of such distributions, called synopses, for the definition of rules that enable similarity searching optimization. An extensive evaluation of Merkurion in real-world datasets has proven its effectiveness and broad applicability to many data domains.

Snaptron: querying splicing patterns across tens of thousands of RNA-seq samples.

MotivationAs more and larger genomics studies appear, there is a growing need for comprehensive and queryable cross-study summaries. These enable researchers to leverage vast datasets that would otherwise be difficult to obtain.ResultsSnaptron is a search engine for summarized RNA sequencing data with a query planner that leverages R-tree, B-tree and inverted indexing strategies to rapidly execute queries over 146 million exon-exon splice junctions from over 70 000 human RNA-seq samples. Queries can be tailored by constraining which junctions and samples to consider. Snaptron can score junctions according to tissue specificity or other criteria, and can score samples according to the relative frequency of different splicing patterns. We describe the software and outline biological questions that can be explored with Snaptron queries.Availability and implementationDocumentation is at http://snaptron.cs.jhu.edu. Source code is at https://github.com/ChristopherWilks/snaptron and https://github.com/ChristopherWilks/snaptron-experiments with a CC BY-NC 4.0 license.Supplementary information Supplementary data are available at Bioinformatics online.

Relaxed operator fusion for in-memory databases

In-memory database management systems (DBMSs) are a key component of modern on-line analytic processing (OLAP) applications, since they provide low-latency access to large volumes of data. Because disk accesses are no longer the principle bottleneck in such systems, the focus in designing query execution engines has shifted to optimizing CPU performance. Recent systems have revived an older technique of using just-in-time (JIT) compilation to execute queries as native code instead of interpreting a plan. The state-of-the-art in query compilation is to fuse operators together in a query plan to minimize materialization overhead by passing tuples efficiently between operators. Our empirical analysis shows, however, that more tactful materialization yields better performance. We present a query processing model called "relaxed operator fusion" that allows the DBMS to introduce staging points in the query plan where intermediate results are temporarily materialized. This allows the DBMS to take advantage of inter-tuple parallelism inherent in the plan using a combination of prefetching and SIMD vectorization to support faster query execution on data sets that exceed the size of CPU-level caches. Our evaluation shows that our approach reduces the execution time of OLAP queries by up to 2.2× and achieves up to 1.8× better performance compared to other in-memory DBMSs.

Efficient generation of query plans containing group-by, join, and groupjoin

It has been a recognized fact for many years that query execution can benefit from pushing grouping operators down in the operator tree and applying them before a join. This so-called eager aggregation reduces the size(s) of the join argument(s), making join evaluation faster. Lately, the idea enjoyed a revival when it was applied to outer joins for the first time and incorporated in a state-of-the-art plan generator. However, the recent approach is highly dependent on the use of heuristics because of the exponential growth of the search space that goes along with eager aggregation. Finding an optimal solution for larger queries calls for effective optimality-preserving pruning mechanisms to reduce the search space size as far as possible. By a more thorough investigation of functional dependencies and keys, we provide a set of new pruning criteria and extend the idea of eager aggregation further by combining it with the introduction of groupjoins. We evaluate the resulting plan generator with respect to runtime and memory consumption.

Query optimization for dynamic imputation

Missing values are common in data analysis and present a usability challenge. Users are forced to pick between removing tuples with missing values or creating a cleaned version of their data by applying a relatively expensive imputation strategy. Our system, ImputeDB, incorporates imputation into a cost-based query optimizer, performing necessary imputations on-the-fly for each query. This allows users to immediately explore their data, while the system picks the optimal placement of imputation operations. We evaluate this approach on three real-world survey-based datasets. Our experiments show that our query plans execute between 10 and 140 times faster than first imputing the base tables. Furthermore, we show that the query results from on-the-fly imputation differ from the traditional base-table imputation approach by 0--8%. Finally, we show that while dropping tuples with missing values that fail query constraints discards 6--78% of the data, on-the-fly imputation loses only 0--21%.

Distributed Top-K Join Queries Optimizing for RDF Datasets

In recent years, the scale of RDF datasets is increasing rapidly, the query research on RDF datasets in the transitional centralized environment is unable to meet the increasing demand of data query field, especially the top-k query. Based on the Spark distributed computing system and the HBase distributed storage system, a novel method is proposed for top-k query. A top–k query plan STA (Spark Threshold Algorithm) is proposed to reduce the connection operation of RDF data. Furthermore, a better algorithm SSJA (Spark Simple Join Algorithm) is presented to reduce the sorting related operations for the intermediate data. A cache mechanism is also proposed to speed up the SSJA algorithm. The experimental results show that the SSJA algorithm performs better than the STA algorithm in term of the cost and applicability, and it can significantly improve the SSJA's performance by introducing the cache mechanism.

Learning to Speed Up Query Planning in Graph Databases

Querying graph structured data is a fundamental operation that enables important applications including knowledge graph search, social network analysis, and cyber-network security. However, the growing size of real-world data graphs poses severe challenges for graph databases to meet the response-time requirements of the applications. Planning the computational steps of query processing — Query Planning — is central to address these challenges. In this paper, we study the problem of learning to speedup query planning in graph databases towards the goal of improving the computational-efficiency of query processing via training queries. We present a Learning to Plan (L2P) framework that is applicable to a large class of query reasoners that follow the Threshold Algorithm (TA) approach. First, we define a generic search space over candidate query plans, and identify target search trajectories (query plans) corresponding to the training queries by performing an expensive search. Subsequently, we learn greedy search control knowledge to imitate the search behavior of the target query plans. We provide a concrete instantiation of our L2P framework for STAR, a state-of-the-art graph query reasoner. Our experiments on benchmark knowledge graphs including dbpedia, yago, and freebase show that using the query plans generated by the learned search control knowledge, we can significantly improve the speed of STAR with negligible loss in accuracy.

Evaluation of Optimization Strategies for Incremental Graph Queries

The last decade brought considerable improvements in distributed storage and query technologies, known as NoSQL systems. These systems provide quick evaluation of simple retrieval operations and are able to answer certain complex queries in a scalable way, albeit not instantly. Providing scalability and quick response times at the same time for querying large data sets is still a challenging task. Evaluating complex graph queries is particularly difficult, as it requires lots of join, antijoin and filtering operations. This paper presents optimization techniques used in relational database systems and applies them on graph queries. We evaluate various query plans on multiple datasets and discuss the effect of different optimization techniques.

A parallel model for SQL astronomical databases based on solid state storage

Abstract Query planning and optimisation algorithms in most popular relational databases were developed at the times hard disk drives were the only storage technology available. The advent of higher parallel random access capacity devices, such as solid state disks, opens up the way for intra-machine parallel computing over large datasets. We describe a two phase parallel model for the implementation of heavy analytical processes in single instance PostgreSQL astronomical databases. This model is particularised to fulfil two frequent astronomical problems, density maps and crossmatch computation with Quad Tree Cube (Q3C) indexes. They are implemented as part of the relational databases infrastructure for the Gaia Archive and performance is assessed. Improvement of a factor 28.40 in comparison to sequential execution is observed in the reference implementation for a histogram computation. Speedup ratios of 3.7 and 4.0 are attained for the reference positional crossmatches considered. We observe large performance enhancements over sequential execution for both CPU and disk access intensive computations, suggesting these methods might be useful with the growing data volumes in Astronomy.

Optimizing Multi-Way Spatial Joins of Web Feature Services

Web Feature Service (WFS) is a widely used spatial web service standard issued by the Open Geospatial Consortium (OGC). In a heterogeneous GIS application, a user can issue a query that relates two or more spatial datasets at different WFS servers. Multi-way spatial joins of WFSs are very expensive in terms of computation and transmission because of the time-consuming interactions between the servers and the client. In this paper, we examine the problems of multi-way spatial joins of WFSs, and we present a client-side optimization approach to generate good execution plans for such queries. The spatial semi-join and area partitioning-based methods are combined to prune away non-candidate objects in processing binary spatial joins, and the filtering rate is used as an index to determine the execution strategy for each sub-area. Two partitioning methods were tested, and the experimental results showed that both are effective if a proper threshold to stop the partitioning is chosen. In processing multi-way spatial joins of WFSs, the filtering rate is used as an indicator to determine the ordering of the binary joins. The optimization method is obviously superior to the other two methods when there are adequate spatial objects involved in the join query, or when more datasets are involved in the join query.

Looking ahead makes query plans robust

Query optimizers and query execution engines cooperate to deliver high performance on complex analytic queries. Typically, the optimizer searches through the plan space and sends a selected plan to the execution engine. However, optimizers may at times miss the optimal plan, with sometimes disastrous impact on performance. In this paper, we develop the notion of robustness of a query evaluation strategy with respect to a space of query plans. We also propose a novel query execution strategy called Lookahead Information Passing (LIP) that is robust with respect to the space of (fully pipeline-able) left-deep query plan trees for in-memory star schema data warehouses. LIP ensures that execution times for the best and the worst case plans are far closer than without LIP. In fact, under certain assumptions of independent and uniform distributions, any plan in that space is theoretically guaranteed to execute in near-optimal time. LIP ensures that the execution time for every plan in the space is nearly-optimal. In this paper, we also evaluate these claims using workloads that include skew and correlation. With LIP we make an initial foray into a novel way of thinking about robustness from the perspective of query evaluation, where we develop strategies (like LIP) that collapse plan sub-spaces in the overall global plan space.

Hadoop에서 SQL 기반 질의언어를 지원하는 공간 빅데이터 질의처리 시스템

본 논문에서는 Hadoop에 공간 데이터를 저장하고 SQL 기반 질의언어에 의하여 공간 데이터를 질의할 수 있는 공간 빅데이터 질의처리 시스템을 제시한다. 제안한 시스템은 대용량의 공간 빅데이터를 HDFS 기반의 저장 시스템에 저장하고 공간 데이터 처리기능이 추가확장된 SQL 기반 질의언어로 질의를 할 수 있도록 지원하며 OGC 심플 피처모델 기반의 공간 데이터 표준 데이터타입과 함수를 지원한다. 본 논문에서는 질의언어 파싱, 질의언어 검증, 질의계획 생성, 저장시스템 연동 등 질의처리의 주요 기능 개발을 제시하였다. 제안한 시스템의 성능을 기존 시스템과 비교하였으며, 실험에서는 Hadoop에 저장된 공간 데이터에 대한 영역질의의 질의실행시간에 있어서 비교 시스템 대비 약 58%의 성능향상을 나타냄을 보였다. In this paper we present a spatial big data query processing system that can store spatial data in Hadoop and query the data with SQL-based query language. The system stores large-scale spatial data in HDFS-based storage system, and supports spatial queries expressed in SQL-based query language extended for spatial data processing. It supports standard spatial data types and functions defined in OGC simple feature model in the query language. This paper presents the development of core functions of the system including query language parsing, query validation, query planning, and connection with storage system. We compares the performance of the suggested system with an existing system, and our experiments show that the system shows about 58% performance improvement of query execution time over the existing system when executing region query for spatial data stored in Hadoop.

A green framework for DBMS based on energy-aware query optimization and energy-efficient query processing

Traditional database systems result in high energy consumption and low energy efficiency due to the lack of consideration of energy issues and environmental adaptation in the design process. In this study, we report our recent efforts on this issue, with a focus on energy-aware query optimization and energy-efficient query processing. Firstly, a method of modeling energy cost of query plans during query processing based on their resource consumption patterns is proposed, which helps predict energy cost of queries before execution. Secondly, as the traditional query optimizer focuses on solely optimizing for performance and ignores energy-efficient query plans, a query-plan evaluation model is proposed after a comprehensive study of plan evaluation principles. Using the cost model as a basis, the evaluation model can utilizes the trade-offs between power and performance of plans, and helps the query optimizer select plans that meet performance requirements but result in lower energy cost. Finally, a green database framework integrated with the two above models is proposed to enhance a commercial DBMS. Experimental results reveal that, with reliable and accurate statistical data, the proposed framework in this study can achieve significant energy savings and improve energy efficiency.

Distributed Query Plan Generation using Bacterial Foraging Optimization

In distributed database systems, relations are replicated and fragmented at multiple sites to ensure easy availability and greater reliability. This leads to an exponential increase in the possible alternatives available for selecting the set of sites, constituting a query plan, for processing. Computing the optimal query plans, from amongst all possible query plans, is a discrete combinatorial optimization problem. This Distributed Query Plan Generation (DQPG) problem has been addressed using Bacterial Foraging Optimization (BFO) in this paper. Here, a novel BFO based DQPG algorithm (DQPGBFO), which generates the Top-K distributed query plans having the minimum total query processing cost, has been proposed. Experimental comparison of DQPGBFO with the existing Genetic Algorithm (GA) based DQPG algorithm (DQPGGA) shows that the former is able to generate Top-K query plans that have a comparatively lower total cost of processing a distributed query. This, in turn, leads to a reduction in the query response time and thus aids in decision making.

Distributed Query Plan Generation using Cuckoo Search Algorithm

Query processing in distributed databases involves data transmission amongst sites capable of providing answers to a distributed query. For this, a distributed query processing strategy, which generates efficient query processing plans for a given distributed query, needs to be devised. Since in distributed databases, the data is fragmented and replicated at multiple sites, the number of query plans increases exponentially with increase in the number of sites capable of providing answers to a distributed query. As a result, generating efficient query processing plans, from amongst all possible query plans, becomes a complex problem. This distributed query plan generation (DQPG) problem has been addressed using the Cuckoo Search Algorithm (CSA) in this paper. Accordingly, a CSA based DQPG algorithm (DQPGCSA) that aims to generate Top-K query plans having minimum cost of processing a distributed query has been proposed. Experimental based comparison of DQPGCSA with the existing GA based DQPG algorithm shows that the former is able to generate Top-K query plans that have a comparatively lower query processing cost. This, in turn, reduces the query response time resulting in efficient decision making.

Logical query optimization for Cloudera Impala system

Abstract Cloudera Impala, an analytic database system for Apache Hadoop, has a severe problem with query plan generation: the system can only generate query plans in left-deep tree form, which restricts the ability of parallel execution. In this paper, we present a logical query optimization scheme for Impala system. First, an improved McCHyp (MinCutConservative Hypergraph) logical query plan generation algorithm is proposed for Impala system. It can reduce the plan generation time by introducing a pruning strategy. Second, a new cost model that takes the characteristics of Impala system into account is proposed. Finally, Impala system is extended to support query plans in bushy tree form by integrating the plan generation algorithm. We evaluated our scheme using TPC-DS test suit. Experimental results show that the extended Impala system generally performs better than the original system, and the improved plan generation algorithm has less execution time than McCHyp. In addition, our cost model fits better for Impala system, which supports query plans in bushy tree form.