Query Execution Engine Research Articles

An overview of the deco system

Deco is a comprehensive system for answering declarative queries posed over stored relational data together with data obtained on-demand from the crowd. In this overview paper, we describe Deco's data model, query language, and system prototype, summarizing material from earlier papers. Deco's data model was designed to be general, flexible, and principled. Deco's query language extends SQL with simple constructs necessary for crowdsourcing, and has a precise semantics for arbitrary queries. Deco's query execution engine and cost-based query optimizer incorporate many novel techniques to address the limitations of traditional query processing techniques in the crowdsourcing setting. Query processing is guided by the objective of minimizing monetary cost and reducing latency.

General Solution to the Requirement-Oriented Integration of Deep Web

In the recent years, the Web has been rapidly “deepened” by the massive web databases on the Internet, which refers to Web”. All the existing methods to access the Deep Web are dataoriented, which don’t consider the various personalized needs of individuals and cannot effectively fill the huge gap between the users and the Deep Web. So the future trend of Deep Web integration must be the systems which are user-oriented and can provide the users with the experience of “fetch on demand”. This paper puts the focus on the topic of requirement-oriented integration of Deep Web and proposes a novel integrate solutions of Deep Web and does some exploratory research on the key issues of this solution. In this solution, the system is composed of three components: Requirement Analysis Engine (RAE), Plan Generation Engine (PGE) and Query Execution Engine (QEE). With the collaborative work of these three engines, the goal of “taking on-demand” and “one-stop” can be achieved. In addition, the open framework of this solution makes it easy to integrate the existing research efforts and techniques.

A Query-based Approach for Semi-Automatic Annotation of Web Services

Semantic Web services (SWS) have attracted increasing attention due to their potential to automate discovery and composition of current syntactic Web services. An issue that prevents a wider adoption of SWS relates to the manual nature of the semantic annotation task. Manual annotation is a difficult, error-prone, and time-consuming process and automating the process is highly desirable. Though some approaches have been proposed to semi-automate the annotation task, they are difficult to use and cannot perform accurate annotation for the following reasons: (1) They require building application ontologies to represent candidate services and (2) they cannot perform accurate name-based matching when labels of candidate service elements and ontological entities contain Compound Nouns (CN). To overcome these two deficiencies, this paper proposes a query-based approach that can facilitate semi-automatic annotation of Web services. The proposed approach is easy to use because it does not require building application ontologies to represent services. Candidate service elements that need to be annotated are extracted from a WSDL file and used to generate query instances by filling a Standard Query Template. The resulting query instances are executed against a repository of ontologies using a novel query execution engine to find appropriate correspondences for candidate service elements. This query execution engine employs name-based and structural matching mechanisms that can perform effective and accurate similarity measurements between labels containing CNs. The proposed semi-automatic annotation approach is evaluated by employing it to annotate existing Web services using published domain ontologies. Precision and recall are used as evaluation metrics. The resulting precision and recall values demonstrate the effectiveness and applicability of the proposed approach.

Sharing-aware horizontal partitioning for exploiting correlations during query processing

Optimization of join queries based on average selectivities is suboptimal in highly correlated databases. In such databases, relations are naturally divided into partitions, each partition having substantially different statistical characteristics. It is very compelling to discover such data partitions during query optimization and create multiple plans for a given query, one plan being optimal for a particular combination of data partitions. This scenario calls for the sharing of state among plans, so that common intermediate results are not recomputed. We study this problem in a setting with a routing-based query execution engine based on eddies [1]. Eddies naturally encapsulate horizontal partitioning and maximal state sharing across multiple plans. We define the notion of a conditional join plan , a novel representation of the search space that enables us to address the problem in a principled way. We present a low-overhead greedy algorithm that uses statistical summaries based on graphical models . Experimental results suggest an order of magnitude faster execution time over traditional optimization for high correlations, while maintaining the same performance for low correlations.

Architecture-Driven Service Discovery for Service Centric Systems

Service discovery has been recognized as an important aspect in the development of service-centric systems, i.e., software systems which deploy Web services. To develop such systems, it is necessary to identify services that can be combined in order to fulfill the functionality and achieve quality criteria of the system being developed. In this paper, we present a framework supporting architecture-driven service discovery (ASD)—that is the discovery of services that can provide functionalities and satisfy properties and constraints of systems as specified during the design phase of the development lifecycle based on detailed system design models. Our framework assumes an iterative design process and allows for the (re-)formulation of design models of service-centric systems based on the discovered services. The framework is composed of a query extractor, which derives queries from behavioral and structural UML design models of service-centric systems, and a query execution engine that executes these queries against service registries based on graph matching techniques. The article describes a prototype tool that we have developed to demonstrate and evaluate our framework and the results of a set of preliminary experiments that we have conducted to evaluate it.

Approximate query processing using wavelets

Approximate query processing has emerged as a cost-effective approach for dealing with the huge data volumes and stringent response-time requirements of today's decision support systems (DSS). Most work in this area, however, has so far been limited in its query processing scope, typically focusing on specific forms of aggregate queries. Furthermore, conventional approaches based on sampling or histograms appear to be inherently limited when it comes to approximating the results of complex queries over high-dimensional DSS data sets. In this paper, we propose the use of multi-dimensional wavelets as an effective tool for general-purpose approximate query processing in modern, high-dimensional applications. Our approach is based on building wavelet-coefficient synopses of the data and using these synopses to provide approximate answers to queries. We develop novel query processing algorithms that operate directly on the wavelet-coefficient synopses of relational tables, allowing us to process arbitrarily complex queries entirely in the wavelet-coefficient domain. This guarantees extremely fast response times since our approximate query execution engine can do the bulk of its processing over compact sets of wavelet coefficients, essentially postponing the expansion into relational tuples until the end-result of the query. We also propose a novel wavelet decomposition algorithm that can build these synopses in an I/O-efficient manner. Finally, we conduct an extensive experimental study with synthetic as well as real-life data sets to determine the effectiveness of our wavelet-based approach compared to sampling and histograms. Our results demonstrate that our techniques: (1) provide approximate answers of better quality than either sampling or histograms; (2) offer query execution-time speedups of more than two orders of magnitude; and (3) guarantee extremely fast synopsis construction times that scale linearly with the size of the data.

The implementation and performance of compressed databases

In this paper, we show how compression can be integrated into a relational database system. Specifically, we describe how the storage manager, the query execution engine, and the query optimizer of a database system can be extended to deal with compressed data. Our main result is that compression can significantly improve the response time of queries if very light-weight compression techniques are used. We will present such light-weight compression techniques and give the results of running the TPC-D benchmark on a so compressed database and a non-compressed database using the AODB database system, an experimental database system that was developed at the Universities of Mannheim and Passau. Our benchmark results demonstrate that compression indeed offers high performance gains (up to 50%) for IO-intensive queries and moderate gains for CPU-intensive queries. Compression can, however, also increase the running time of certain update operations. In all, we recommend to extend today's database systems with light-weight compression techniques and to make extensive use of this feature.

Query processing in DOQL: A deductive database language for the ODMG model

This paper describes the architecture, algebraic query processing framework and query execution approach that comprise the implementation of the deductive object query language (DOQL) query processing system. To the best of our knowledge, it is the first deductive object query language to be designed and implemented as a complementary and non-intrusive query component within an ODMG OODBMS architecture. The query processing framework enables the combined use of logical rewriting and algebraic optimization, and features an object algebra, local and global query optimization, physical execution algorithms implemented as iterators, and a query execution engine implemented using the dataflow technique. Several representative DOQL queries are also provided, illustrating the flexibility and expressiveness of querying object databases with DOQL.

Self-tuning histograms

In this paper, we introduce self-tuning histograms. Although similar in structure to traditional histograms, these histograms infer data distributions not by examining the data or a sample thereof, but by using feedback from the query execution engine about the actual selectivity of range selection operators to progressively refine the histogram. Since the cost of building and maintaining self-tuning histograms is independent of the data size, self-tuning histograms provide a remarkably inexpensive way to construct histograms for large data sets with little up-front costs. Self-tuning histograms are particularly attractive as an alternative to multi-dimensional traditional histograms that capture dependencies between attributes but are prohibitively expensive to build and maintain. In this paper, we describe the techniques for initializing and refining self-tuning histograms. Our experimental results show that self-tuning histograms provide a low-cost alternative to traditional multi-dimensional histograms with little loss of accuracy for data distributions with low to moderate skew.

Sort vs. hash revisited

Efficient algorithms for processing large volumes of data are very important both for relational and new object-oriented database systems. Many query-processing operations can be implemented using sort- or hash-based algorithms, e.g. intersections, joins, and duplicate elimination. In the early relational database systems, only sort-based algorithms were employed. In the last decade, hash-based algorithms have gained acceptance and popularity, and are often considered generally superior to sort-based algorithms such as merge-join. In this article, we compare the concepts behind sort- and hash-based query-processing algorithms and conclude that (1) many dualities exist between the two types of algorithms, (2) their costs differ mostly by percentages rather than by factors, (3) several special cases exist that favor one or the other choice, and (4) there is a strong reason why both hash- and sort-based algorithms should be available in a query-processing system. Our conclusions are supported by experiments performed using the Volcano query execution engine.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Volcano-an extensible and parallel query evaluation system

To investigate the interactions of extensibility and parallelism in database query processing, we have developed a new dataflow query execution system called Volcano. The Volcano effort provides a rich environment for research and education in database systems design, heuristics for query optimization, parallel query execution, and resource allocation. Volcano uses a standard interface between algebra operators, allowing easy addition of new operators and operator implementations. Operations on individual items, e.g., predicates, are imported into the query processing operators using support functions. The semantics of support functions is not prescribed; any data type including complex objects and any operation can be realized. Thus, Volcano is extensible with new operators, algorithms, data types, and type-specific methods. Volcano includes two novel meta-operators. The choose-plan meta-operator supports dynamic query evaluation plans that allow delaying selected optimization decisions until run-time, e.g., for embedded queries with free variables. The exchange meta-operator supports intra-operator parallelism on partitioned datasets and both vertical and horizontal inter-operator parallelism, translating between demand-driven dataflow within processes and data-driven dataflow between processes. All operators, with the exception of the exchange operator, have been designed and implemented in a single-process environment, and parallelized using the exchange operator. Even operators not yet designed can be parallelized using this new operator if they use and provide the interator interface. Thus, the issues of data manipulation and parallelism have become orthogonal, making Volcano the first implemented query execution engine that effectively combines extensibility and parallelism.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Encapsulation of parallelism and architecture-independence in extensible database query execution

Emerging database application domains demand not only high functionality, but also high performance. To satisfy these two requirements, the Volcano query execution engine combines the efficient use of parallelism on a wide variety of computer architectures with an extensible set of query processing operators that can be nested into arbitrarily complex query evaluation plans. Volcano's novel exchange operator permits designing, developing, debugging, and tuning data manipulation operators in single-process environments but executing them in various forms of parallelism. The exchange operator shields the data manipulation operators from all parallelism issues. The design and implementation of the generalized exchange operator are examined. The authors justify their decision to support hierarchical architectures and argue that the exchange operator offers a significant advantage for development and maintenance of database query processing software. They discuss the integration of bit vector filtering into the exchange operator paradigm with only minor modifications. >