Query Evaluation Algorithms Research Articles

Relational Algorithms for Top-k Query Evaluation

The evaluation of top-k conjunctive queries, a staple in business analysis, often requires evaluating the conjunctive query prior to filtering the top-k results, leading to a significant computational overhead within Database Management Systems (DBMSs). While efficient algorithms have been proposed, their integration into DBMSs remains arduous. We introduce relational algorithms, a paradigm where each algorithmic step is expressed by a relational operator. This allows the algorithm to be represented as a set of SQL queries, enabling easy deployment across different systems that support SQL. We introduce two novel relational algorithms, level-k and product-k, specifically designed for evaluating top-k conjunctive queries and demonstrate that level-k achieves optimal running time for top-k free-connex queries. Furthermore, these algorithms enable easy translation into an oblivious algorithm for secure query evaluations. The presented algorithms are not only theoretically optimal but also exhibit eminent efficiency in practice. The experiment results show significant improvements, with our rewritten SQL outperforming the baseline by up to 6 orders of magnitude. Moreover, our secure implementations not only achieve substantial speedup compared to the baseline with secure guarantees but even surpass those baselines that have no secure guarantees.

Efficient query evaluation techniques over large amount of distributed linked data

As RDF becomes more widely established and the amount of linked data is rapidly increasing, the efficient querying of large amount of data becomes a significant challenge. In this paper, we propose a family of algorithms for querying large amount of linked data in a distributed manner. These query evaluation algorithms are independent of the way the data is stored, as well as of the particular implementation of the query evaluation. We then use the MapReduce paradigm to present a distributed implementation of these algorithms and experimentally evaluate them, although the algorithms could be straightforwardly translated into other distributed processing frameworks. We also investigate and propose multiple query decomposition approaches of Basic Graph Patterns (subclass of SPARQL queries) that are used to improve the overall performance of the distributed query answering. A deep analysis of the effectiveness of these decomposition algorithms is also provided.

Threshold queries in theory and in the wild

Threshold queries are an important class of queries that only require computing or counting answers up to a specified threshold value. To the best of our knowledge, threshold queries have been largely disregarded in the research literature, which is surprising considering how common they are in practice. In this paper, we present a deep theoretical analysis of threshold query evaluation and show that thresholds can be used to significantly improve the asymptotic bounds of state-of-the-art query evaluation algorithms. We also empirically show that threshold queries are significant in practice. In surprising contrast to conventional wisdom, we found important scenarios in real-world data sets in which users are interested in computing the results of queries up to a certain threshold, independent of a ranking function that orders the query results.

Proactive and intelligent evaluation of big data queries in edge clouds with materialized views

The rise of big data brings extraordinary benefits and opportunities to businesses and governments. Enterprise users can analyze their consumers’ data and infer the business value, such as purchasing goods correlations and customer preferences. To unveil such hidden values from big data, various big data processing frameworks, such as Hadoop and Tensor-flow, are developed. One fundamental approach of accelerating such big data processing frameworks is to efficiently evaluate queries for big data analytics based on materialization of intermediate results. In this paper, we consider problems of dynamic and proactive QoS-aware query evaluation of big data analytics with intermediate result materialization in a mobile edge cloud. We propose a one-shot online algorithm for the dynamic QoS-aware big data query evaluation within a finite time horizon, which can intelligently determine whether some immediate results during query evaluation need to be materialized for later use of other queries, by making use of the Reinforcement Learning (RL) method with predictions. We also devise an online learning algorithm based on the technique of multi-armed bandits, for the proactive QoS-aware big data query evaluation problem with resource reservations, where the query arrival information and their required datasets are uncertain before the actual evaluation of the queries. We finally investigate the performance of the proposed algorithms by experimental simulations, and results show that the performance of the proposed algorithms is promising, by achieving 12% higher system throughput while reducing 50% and 40% average evaluation cost and time per query compared to the comparison benchmarks.

A Query Language for Workflow Logs

A business process (workflow) is an assembly of tasks to accomplish a business goal. Real-world workflow models often demanded to change due to new laws and policies, changes in the environment, and so on. To understand the inner workings of a business process to facilitate changes, workflow logs have the potential to enable inspecting, monitoring, diagnosing, analyzing, and improving the design of a complex workflow. Querying workflow logs, however, is still mostly an ad hoc practice by workflow managers. In this article, we focus on the problem of querying workflow log concerning both control flow and dataflow properties. We develop a query language based on “incident patterns” to allow the user to directly query workflow logs instead of having to transform such queries into database operations. We provide the formal semantics and a query evaluation algorithm of our language. By deriving an accurate cost model, we develop an optimization mechanism to accelerate query evaluation. Our experiment results demonstrate the effectiveness of the optimization and achieves up to 50× speedup over an adaption of existing evaluation method.

On Completeness Classes for Query Evaluation on Linked Data

The advent of the Web of Data kindled interest in link-traversal (or lookup-based) query processing methods, with which queries are answered via dereferencing a potentially large number of small, interlinked sources. While several algorithms for query evaluation have been proposed, there exists no notion of completeness for results of so-evaluated queries. In this paper, we motivate the need for clearly-defined completeness classes and present several notions of completeness for queries over Linked Data, based on the idea of authoritativeness of sources, and show the relation between the different completeness classes.

Context-free path querying with all-path semantics using matrices with sets of intermediate vertices

The study considers the problem of context-free path querying with all-path query semantics. This problem consists in finding all paths of the graph, the labels on the edges of which form words from the language generated by the input context-free grammar. There are two approaches to evaluate context-free path queries using linear algebra operations: matrix multiplication-based and the Kronecker product-based. But until now, there is no algorithm using the matrix multiplication capable of handling context-free path queries with the most complex all-path query semantics, in which the all paths that match the query must be provided. The paper proposes the algorithm for context-free path query evaluation using the matrix multiplication, which is capable of processing queries with the all-path query semantics. In the adjacency matrix of the input graph for each pair of vertices, we store additional information about the paths found between these vertices in the form of a set of possible intermediate vertices. At the first stage, a set of matrices is constructed that store such information about all paths that satisfy the input query. At the second stage, all queried paths are restored from the constructed set of matrices. The proposed algorithm was implemented in C++ and a comparison was made with other most efficient algorithms for evaluating context-free path queries, namely with the matrix-based algorithm that allows us to find only one such path, and with the Kronecker product-based algorithm that allows us to find all such paths in the graph. The results of the experimental study showed that the proposed algorithm is significantly more efficient in restoring the queried paths, but in some cases it consumes a significantly larger amount of memory than the algorithm based on the Kronecker product. The described algorithm can be applied in static code analysis, bioinformatics, network analysis, as well as in graph databases, when it is required to find all possible dependencies in the data presented in the form of a labeled graph.

Efficient Authorization of Graph-database Queries in an Attribute-supporting ReBAC Model

Neo4j is a popular graph database that offers two versions: an enterprise edition and a community edition . The enterprise edition offers customizable Role-based Access Control features through custom developed procedures , while the community edition does not offer any access control support. Being a graph database, Neo4j appears to be a natural application for Relationship-Based Access Control (ReBAC), an access control paradigm where authorization decisions are based on relationships between subjects and resources in the system (i.e., an authorization graph). In this article, we present AReBAC, an attribute-supporting ReBAC model for Neo4j that provides finer-grained access control by operating over resources instead of procedures. AReBAC employs Nano-Cypher, a declarative policy language based on Neo4j’s Cypher query language, the result of which allows us to weave database queries with access control policies and evaluate both simultaneously. Evaluating the combined query and policy produces a result that (i) matches the search criteria, and (ii) the requesting subject is authorized to access. AReBAC is accompanied by the algorithms and their implementation required for the realization of the presented ideas, including GP-Eval, a query evaluation algorithm. We also introduce Live-End Backjumping (LBJ), a backtracking scheme that provides a significant performance boost over conflict-directed backjumping for evaluating queries. As demonstrated in our previous work, the original version of GP-Eval already performs significantly faster than the Neo4j’s Cypher evaluation engine. The optimized version of GP-Eval , which employs LBJ, further improves the performance significantly, thereby demonstrating the capabilities of the technique.

Learning Models over Relational Data Using Sparse Tensors and Functional Dependencies

Integrated solutions for analytics over relational databases are of great practical importance as they avoid the costly repeated loop data scientists have to deal with on a daily basis: select features from data residing in relational databases using feature extraction queries involving joins, projections, and aggregations; export the training dataset defined by such queries; convert this dataset into the format of an external learning tool; and train the desired model using this tool. These integrated solutions are also a fertile ground of theoretically fundamental and challenging problems at the intersection of relational and statistical data models. This article introduces a unified framework for training and evaluating a class of statistical learning models over relational databases. This class includes ridge linear regression, polynomial regression, factorization machines, and principal component analysis. We show that, by synergizing key tools from database theory such as schema information, query structure, functional dependencies, recent advances in query evaluation algorithms, and from linear algebra such as tensor and matrix operations, one can formulate relational analytics problems and design efficient (query and data) structure-aware algorithms to solve them. This theoretical development informed the design and implementation of the AC/DC system for structure-aware learning. We benchmark the performance of AC/DC against R, MADlib, libFM, and TensorFlow. For typical retail forecasting and advertisement planning applications, AC/DC can learn polynomial regression models and factorization machines with at least the same accuracy as its competitors and up to three orders of magnitude faster than its competitors whenever they do not run out of memory, exceed 24-hour timeout, or encounter internal design limitations.

Distributed graph queries over models@run.time for runtime monitoring of cyber-physical systems

Smart cyber-physical systems (CPSs) have complex interaction with their environment which is rarely known in advance, and they heavily depend on intelligent data processing carried out over a heterogeneous and distributed computation platform with resource-constrained devices to monitor, manage and control autonomous behavior. First, we propose a distributed runtime model to capture the operational state and the context information of a smart CPS using directed, typed and attributed graphs as high-level knowledge representation. The runtime model is distributed among the participating nodes, and it is consistently kept up to date in a continuously evolving environment by a time-triggered model management protocol. Our runtime models offer a (domain-specific) model query and manipulation interface over the reliable communication middleware of the Data Distribution Service (DDS) standard widely used in the CPS domain. Then, we propose to carry out distributed runtime monitoring by capturing critical properties of interest in the form of graph queries, and design a distributed graph query evaluation algorithm for evaluating such graph queries over the distributed runtime model. As the key innovation, our (1) distributed runtime model extends existing publish–subscribe middleware (like DDS) used in real-time CPS applications by enabling the dynamic creation and deletion of graph nodes (without compile time limits). Moreover, (2) our distributed query evaluation extends existing graph query techniques by enabling query evaluation in a real-time, resource-constrained environment while still providing scalable performance. Our approach is illustrated, and an initial scalability evaluation is carried out on the MoDeS3 CPS demonstrator and the open Train Benchmark for graph queries.

MWAND: A New Early Termination Algorithm for Fast and Efficient Query Evaluation

MWAND: A New Early Termination Algorithm for Fast and Efficient Query Evaluation

Querying Workflow Logs

A business process or workflow is an assembly of tasks that accomplishes a business goal. Business process management is the study of the design, configuration/implementation, enactment and monitoring, analysis, and re-design of workflows. The traditional methodology for the re-design and improvement of workflows relies on the well-known sequence of extract, transform, and load (ETL), data/process warehousing, and online analytical processing (OLAP) tools. In this paper, we study the ad hoc queryiny of process enactments for (data-centric) business processes, bypassing the traditional methodology for more flexibility in querying. We develop an algebraic query language based on “incident patterns” with four operators inspired from Business Process Model and Notation (BPMN) representation, allowing the user to formulate ad hoc queries directly over workflow logs. A formal semantics of this query language, a preliminary query evaluation algorithm, and a group of elementary properties of the operators are provided.

Data driven approximation with bounded resources

This paper proposes BEAS, a resource-bounded scheme for querying relations. It is parameterized with a resource ratio α ∈ (0,1], indicating that given a big dataset D , we can only afford to access an α -fraction of D with limited resources. For a query Q posed on D , BEAS computes exact answers Q(D) if doable and otherwise approximate answers, by accessing at most α | D | amount of data in the entire process. Underlying BEAS are (1) an access schema, which helps us identify and fetch the part of data needed to answer Q , (2) an accuracy measure to assess approximate answers in terms of their relevance and coverage w.r.t . exact answers, (3) an Approximability Theorem for the feasibility of resource-bounded approximation, and (4) algorithms for query evaluation with bounded resources. A unique feature of BEAS is its ability to answer unpredictable queries, aggregate or not, using bounded resources and assuring a deterministic accuracy lower bound. Using real-life and synthetic data, we empirically verify the effectiveness and efficiency of BEAS.

Ontology assisted crowd mining

We present OASSIS (for Ontology ASSISted crowd mining), a prototype system which allows users to declaratively specify their information needs, and mines the crowd for answers. The answers that the system computes are concise and relevant , and represent frequent, significant data patterns. The system is based on (1) a generic model that captures both ontological knowledge, as well as the individual knowledge of crowd members from which frequent patterns are mined; (2) a query language in which users can specify their information needs and types of data patterns they seek; and (3) an efficient query evaluation algorithm, for mining semantically concise answers while minimizing the number of questions posed to the crowd. We will demonstrate OASSIS using a couple of real-life scenarios, showing how users can formulate and execute queries through the OASSIS UI and how the relevant data is mined from the crowd.

Алгоритмы обработки запросов к дедуктивным базам данных и реализация алгоритма QSQ

Алгоритмы обработки запросов к дедуктивным базам данных и реализация алгоритма QSQ

Energy-efficient top-k query evaluation and maintenance in wireless sensor networks

Top-k query in a wireless sensor network is to identify k sensors with the highest sensor readings. Since sensors usually are powered by energy-limited batteries, a fundamental problem associated with top-k query evaluation in such a network is to maximize network lifetime, which poses great challenges due to the unique characteristics of sensor networks. In this paper, we first propose a novel filter-based algorithm for top-k query evaluation, which is able to filter out a fractional amount of data from network-wide transmission. We then develop an online algorithm for answering time-dependent top-k queries with different values of k through the dynamic maintenance of a materialized view that consists of historical top-k results. We finally conduct extensive experiments by simulations to evaluate the performance of the proposed algorithms using real sensing data sets. Experimental results show that the proposed algorithms outperform a well known existing algorithm significantly, and the network lifetime delivered by the proposed optimal quantile algorithm is at least 142 % times longer than that by an existing algorithm.

Query Evaluation for XML Databases using Partial Decompression

This research work demonstrates the Extraction, compression and query processing of XML documents for Adaptive Compression Techniques and Efficient Query Evaluation. Proposed here are the algorithms for xml compression and Efficient Query Evaluation as - Feasible XML compression using data compression algorithm. Qurey Processor using Sax parsing and Interfaces. It is shown that using the proposed techniques for xml data compression will pave a way for better compression and improve the compression ratio and performance of compressor system

Top-k queries over web applications

The core logic of web applications that suggest some particular service, such as online shopping, e-commerce etc., is typically captured by Business Processes (BPs). Among all the (maybe infinitely many) possible execution flows of a BP, analysts are often interested in identifying flows that are "most important", according to some weight metric. The goal of the present paper is to provide efficient algorithms for top-k query evaluation over the possible executions of Business Processes, under some given weight function. Unique difficulties in top-k analysis in this settings stem from (1) the fact that the number of possible execution flows of a given BP is typically very large, or even infinite in presence of recursion and (2) that the weights (e.g., likelihood, monetary cost, etc.) induced by actions performed during the execution (e.g., product purchase) may be inter-dependent (due to probabilistic dependencies, combined discount deals etc.). We exemplify these difficulties, and overcome them to provide efficient algorithms for query evaluation where possible. We also describe in details an application prototype that we have developed for recommending optimal navigation in an online shopping web site that is based on our model and algorithms.

An Algorithm to Evaluate Iceberg Query using Compacted Bitmap Vector

The data storing and retrieving are playing a major role in the data clustering and data warehousing techniques. The effectiveness of a data retrieving method depends upon the data specific queries for retrieving the data from the database. Iceberg query is a unique class of aggregation query, which computes aggregate values above a given threshold. Many data mining queries are basically ice berg queries. The major part taken into the consideration about the AND operation in the iceberg queries. The reduced number of AND operation increases the effectiveness of the iceberg query. In this work, an efficient iceberg query evaluation process is proposed by reducing the bitwise AND operations needed to find the item pairs. In the proposed approach two solutions are introduced to reduce the bitwise AND operations. Randomly identifying 'N' 1-bit positions instead of first 1-bit position and reducing the zero bit values from the Most Significant Side so that the bit map vector will be reduced in such a way, the bitwise operations needed is reduced. The experimentation is conducted on two datasets in order to evaluate the performance of the proposed iceberg query evaluation algorithm. In the case of retail datasets, the time required for processing 100000 tuples is 864 milliseconds. On the other hand, the same for synthetic dataset, T10I4D100K, is 910 milliseconds.

ANDES: efficient evaluation of NOT-twig queries in relational databases

Despite a large body of work on XPath query processing in relational environment, systematic study of queries containing not-predicates have received little attention in the literature. Particularly, several xml supports of industrial-strength commercial rdbms fail to efficiently evaluate such queries. In this paper, we present an efficient and novel strategy to evaluate not -twig queries in a tree-unaware relational environment. not -twig queries are XPath queries with ancestor---descendant and parent---child axis and contain one or more not-predicates. We propose a novel Dewey-based encoding scheme called Andes (ANcestor Dewey-based Encoding Scheme), which enables us to efficiently filter out elements satisfying a not-predicate by comparing their ancestor group identifiers. In this approach, a set of elements under the same common ancestor at a specific level in the xml tree is assigned same ancestor group identifier. Based on this scheme, we propose a novel sql translation algorithm for not-twig query evaluation. Experiments carried out confirm that our proposed approach built on top of an off-the-shelf commercial rdbms significantly outperforms state-of-the-art relational and native approaches. We also explore the query plans selected by a commercial relational optimizer to evaluate our translated queries in different input cardinality. Such exploration further validates the performance benefits of Andes.