Aggregate Queries Research Articles

Online approximative SPARQL query processing for COUNT-DISTINCT queries with web preemption

Getting complete results when processing aggregate queries on public SPARQL endpoints is challenging, mainly due to the application of quotas. Although Web preemption supports processing of aggregate queries online, on preemptable SPARQL servers, data transfer is still very large when processing count-distinct aggregate queries. In this paper, it is shown that count-distinct aggregate queries can be approximated with low data transfer by extending the partial aggregation operator with HyperLogLog++ sketches. Experimental results demonstrate that the proposed approach outperforms existing approaches by orders of magnitude in terms of the amount of data transferred.

Comparison of Data Sampling Strategies for Approximate Processing of Queries to a Large Database

Interactive analytics requires database systems to promptly process aggregation queries. As the volume of data continues to grow at a high rate, this becomes more and more challenging. In the past, the database community has come up with two ideas to solve this problem: Approximate Query Processing (AQP) and Aggregate Precomputation (AggPre). In this article, we will focus on APQ. Many techniques were proposed for optimizing AQP performance. Most of them are based on the creation of the best stratified samples. But none of the existing approaches can answer the following questions: "Under what conditions will the sampling strategy be optimal? What is the maximum error in estimating an aggregate when using the strategy in case of data skew?" The article provides answers to the questions posed. The article discusses three strategies for selecting segments from a data array: 1) uniform, 2) based on the distribution of read records by segments, 3) based on the distribution of read records and aggregate values by segments. The article defines conditions for the optimality of each strategy. Proposed analytical methods solve the optimization problem and determine the conditions under which the value of the confidence interval of the estimated aggregate will be minimal. The paper also studied the effect of data skews on the accuracy of the estimation of aggregated values. The maximum value of the confidence interval is calculated at the border of the admissible region. Experiments on depersonalized banMng transactions demonstrated that the 3rd sampling strategy is much better than the first two strategies.

Towards a Grid-Based Framework for Supporting Range Aggregate Queries Over Big Sensor Network Readings

The problem of representing and querying sensor network readings issues new research challenges, as traditional techniques and architectures used for managing relational and object-oriented databases are not suitable in this context. In this paper, the authors present a grid-based framework that supports aggregate query answering on sensor network data and uses a summarization technique to efficiently accomplish this task. In particular, grid nodes are used for collecting, compressing, and storing sensor network readings, as well as extracting information from stored data. Grid nodes can exchange information among each other, so that the same piece of information can be stored (with a different degree of accuracy) in several nodes. Queries are evaluated by locating the grid nodes containing the needed information (either compressed or not) and choosing (among these nodes) the most convenient ones according to a cost model. The authors complete their contribution with a case study that focuses attention on the management and querying of grid-based GIS databases.

Infinite Probabilistic Databases

Probabilistic databases (PDBs) model uncertainty in data in a quantitative way. In the established formal framework, probabilistic (relational) databases are finite probability spaces over relational database instances. This finiteness can clash with intuitive query behavior (Ceylan et al., KR 2016), and with application scenarios that are better modeled by continuous probability distributions (Dalvi et al., CACM 2009). We formally introduced infinite PDBs in (Grohe and Lindner, PODS 2019) with a primary focus on countably infinite spaces. However, an extension beyond countable probability spaces raises nontrivial foundational issues concerned with the measurability of events and queries and ultimately with the question whether queries have a well-defined semantics. We argue that finite point processes are an appropriate model from probability theory for dealing with general probabilistic databases. This allows us to construct suitable (uncountable) probability spaces of database instances in a systematic way. Our main technical results are measurability statements for relational algebra queries as well as aggregate queries and Datalog queries.

Compression Methodologies for Columnar Database Optimization

Today’s life is completely dependent on data. Conventional relational databases take longer to respond to queries because they are built for row-wise data storage and retrieval. Due to their efficient read and write operations to and from hard discs, which reduce the time it takes for queries to produce results, columnar databases have recently overtaken traditional databases. To execute Business Intelligence and create decision-making systems, vast amounts of data gathered from various sources are required in data warehouses, where columnar databases are primarily created. Since the data are stacked closely together, and the seek time is reduced, columnar databases perform queries more quickly. With aggregation queries to remove unnecessary data, they allow several compression techniques for faster data access. To optimise the efficiency of columnar databases, various compression approaches, including NULL Suppression, Dictionary Encoding, Run Length Encoding, Bit Vector Encoding, and Lempel Ziv Encoding, are discussed in this work. Database operations are conducted on the compressed data to demonstrate the decrease in memory needs and speed improvements.

A Gitbash-Generative Model for an Enhanced Query Processing In a Distributed Database System

This research study is directly focused on an enhanced query processing system using GitBash- Deep Generative Model, it is use to improve the performance of big data. Due to Poor query execution plan and absence of an improved aggregate query method which impose threats to qualities of good query processing in distributed databases which also cause memory pressure in a database, inflated central processing unit (CPU) and an overall reduction in concurrency. Also the volume of data use in any organization is progressively increasing in it seize, thereby resulting or demand a large storage location or space system. The availability of intended software that will confront the problem of storage contents and execution plan for query processing is a vital challenge facing big data in a processing system, which can prompt to information loss and missing record of vital document in an enterprises. The study will boost solution and addresses the problem of latencies, inability to transform and store queries, and poor query processing plans for future improvement of distributed databases, using an unsupervised learning approach such as deep generative model. The adopted methodology is dynamic system development method (DSDM). php programming language and mongoDB database is adopted for implementation. The datasets used for the proposed system was generated from e-library server/data.json (query) and is inserted into MongoDB database json format. The datasets are all trained and recognized by the Deep Generative algorithm through the Git Bash server. The proposed system result shows highly increase in the efficiency of management of big data and improved query processing. General Term Query Processing System Keywords Big Data, Deep Generative Model, Query Processing System, Information Management, Gitbash.

Exploiting Reused-Based Sharing Work Opportunities in Big Data Multiquery Optimization with Flink.

Multiquery optimization is fundamental to retrieve data from different sources in a specific time frame to fulfill the credibility of the big data applications. To avoid high-cost inputs or outputs operations over large-scale data, exploiting sharing opportunities including (join, aggregation, and sort) is helpful to improve the performance of the multiple queries. Furthermore, considering the in-memory big data platforms such as Flink is essential to enhance the performance of the multiple queries. We have extended our previous proposed system, Multi-Query Optimization using Tuple Size and Histogram (MOTH), to exploit sharing work among multiquery, including join, aggregation, and sort. The comprehensive system, called the Join-Aggregation-Sort (JAS)-MOTH system, is used to minimize the data in-network movement time, that is, shuffle time needed to transfer intermediate data. The proposed system is introduced by developing two additional modules to investigate the sharing work; query explorer and JAS-MOTH optimizer, including the sort exploiter module. The JAS-MOTH system can exploit the shared explicit and implicit sorts among multiple sorts and aggregation queries. The proposed system refines the pipelined multiway join execution for multiple join queries by considering coarse-grained sharing data, join ordering, joining pipelining, and shared implicit sorts. Furthermore, it introduces an end-to-end multiway join optimizer over Flink. An elaborated experimental comparison of the naive and state-of-art techniques among a broader class of queries (i.e., join, aggregation, and sort) is presented. Given the consideration of the findings based on Flink, the proposed system has improved the query execution time regarding the naive and state-of-art techniques by 47% and 30%, respectively. For intermediate data size, the JAS-MOTH system has reduced the intermediate data size reduction concerning the naive and state-of-art techniques by 50% and 31% on average, respectively, over Hadoop-like infrastructures.

The Shapley Value of Tuples in Query Answering

We investigate the application of the Shapley value to quantifying the contribution of a tuple to a query answer. The Shapley value is a widely known numerical measure in cooperative game theory and in many applications of game theory for assessing the contribution of a player to a coalition game. It has been established already in the 1950s, and is theoretically justified by being the very single wealth-distribution measure that satisfies some natural axioms. While this value has been investigated in several areas, it received little attention in data management. We study this measure in the context of conjunctive and aggregate queries by defining corresponding coalition games. We provide algorithmic and complexity-theoretic results on the computation of Shapley-based contributions to query answers; and for the hard cases we present approximation algorithms.

On detecting cherry-picked generalizations

Generalizing from detailed data to statements in a broader context is often critical for users to make sense of large data sets. Correspondingly, poorly constructed generalizations might convey misleading information even if the statements are technically supported by the data. For example, a cherry-picked level of aggregation could obscure substantial sub-groups that oppose the generalization. We present a framework for detecting and explaining cherry-picked generalizations by refining aggregate queries. We present a scoring method to indicate the appropriateness of the generalizations. We design efficient algorithms for score computation. For providing a better understanding of the resulting score, we also formulate practical explanation tasks to disclose significant counterexamples and provide better alternatives to the statement. We conduct experiments using real-world data sets and examples to show the effectiveness of our proposed evaluation metric and the efficiency of our algorithmic framework.

Database Principles and Challenges in Text Analysis

A common conceptual view of text analysis is that of a two-step process, where we first extract relations from text documents and then apply a relational query over the result. Hence, text analysis shares technical challenges with, and can draw ideas from, relational databases. A framework that formally instantiates this connection is that of the document spanners. In this article, we review recent advances in various research efforts that adapt fundamental database concepts to text analysis through the lens of document spanners. Among others, we discuss aspects of query evaluation, aggregate queries, provenance, and distributed query planning.

Accelerating approximate aggregation queries with expensive predicates

Researchers and industry analysts are increasingly interested in computing aggregation queries over large, unstructured datasets with selective predicates that are computed using expensive deep neural networks (DNNs). As these DNNs are expensive and because many applications can tolerate approximate answers, analysts are interested in accelerating these queries via approximations. Unfortunately, standard approximate query processing techniques to accelerate such queries are not applicable because they assume the result of the predicates are available ahead of time. Furthermore, recent work using cheap approximations (i.e., proxies) do not support aggregation queries with predicates. To accelerate aggregation queries with expensive predicates, we develop and analyze a query processing algorithm that leverages proxies (ABAE). ABAE must account for the key challenge that it may sample records that do not satisfy the predicate. To address this challenge, we first use the proxy to group records into strata so that records satisfying the predicate are ideally grouped into few strata. Given these strata, ABAE uses pilot sampling and plugin estimates to sample according to the optimal allocation. We show that ABAE converges at an optimal rate in a novel analysis of stratified sampling with draws that may not satisfy the predicate. We further show that ABAE outperforms on baselines on six real-world datasets, reducing labeling costs by up to 2.3X.

Explaining inference queries with bayesian optimization

Obtaining an explanation for an SQL query result can enrich the analysis experience, reveal data errors, and provide deeper insight into the data. Inference query explanation seeks to explain unexpected aggregate query results on inference data; such queries are challenging to explain because an explanation may need to be derived from the source, training, or inference data in an ML pipeline. In this paper, we model an objective function as a black-box function and propose BOExplain, a novel framework for explaining inference queries using Bayesian optimization (BO). An explanation is a predicate defining the input tuples that should be removed so that the query result of interest is significantly affected. BO --- a technique for finding the global optimum of a black-box function --- is used to find the best predicate. We develop two new techniques (individual contribution encoding and warm start) to handle categorical variables. We perform experiments showing that the predicates found by BOExplain have a higher degree of explanation compared to those found by the state-of-the-art query explanation engines. We also show that BOExplain is effective at deriving explanations for inference queries from source and training data on a variety of real-world datasets. BOExplain is open-sourced as a Python package at https://github.com/sfu-db/BOExplain.

An intermediate representation for hybrid database and machine learning workloads

IFAQ is an intermediate representation and compilation framework for hybrid database and machine learning workloads expressible using iterative programs with functional aggregate queries. We demonstrate IFAQ for several OLAP queries, linear algebra expressions, and learning factorization machines over training datasets defined by feature extraction queries over relational databases.

Query Games in Databases

Database tuples can be seen as players in the game of jointly realizing the answer to a query. Some tuples may contribute more than others to the outcome, which can be a binary value in the case of a Boolean query, a number for a numerical aggregate query, and so on. To quantify the contributions of tuples, we use the Shapley value that was introduced in cooperative game theory and has found applications in a plethora of domains. Specifically, the Shapley value of an individual tuple quantifies its contribution to the query. We investigate the applicability of the Shapley value in this setting, as well as the computational aspects of its calculation in terms of complexity, algorithms, and approximation.

Multi-Dimensional Event Data in Graph Databases

Process event data is usually stored either in a sequential process event log or in a relational database. While the sequential, single-dimensional nature of event logs aids querying for (sub)sequences of events based on temporal relations such as “directly/eventually-follows,” it does not support querying multi-dimensional event data of multiple related entities. Relational databases allow storing multi-dimensional event data, but existing query languages do not support querying for sequences or paths of events in terms of temporal relations. In this paper, we propose a general data model for multi-dimensional event data based on labeled property graphs that allows storing structural and temporal relations in a single, integrated graph-based data structure in a systematic way. We provide semantics for all concepts of our data model, and generic queries for modeling event data over multiple entities that interact synchronously and asynchronously. The queries allow for efficiently converting large real-life event data sets into our data model, and we provide 5 converted data sets for further research. We show that typical and advanced queries for retrieving and aggregating such multi-dimensional event data can be formulated and executed efficiently in the existing query language Cypher, giving rise to several new research questions. Specifically, aggregation queries on our data model enable process mining over multiple inter-related entities using off-the-shelf technology.

Resource Sharing Scheme for Continuous Aggregation Query Processing in Spark Frameworks

Resource Sharing Scheme for Continuous Aggregation Query Processing in Spark Frameworks

Scotty

Window aggregation is a core operation in data stream processing. Existing aggregation techniques focus on reducing latency, eliminating redundant computations, or minimizing memory usage. However, each technique operates under different assumptions with respect to workload characteristics, such as properties of aggregation functions (e.g., invertible, associative), window types (e.g., sliding, sessions), windowing measures (e.g., time- or count-based), and stream (dis)order. In this article, we present Scotty , an efficient and general open-source operator for sliding-window aggregation in stream processing systems, such as Apache Flink, Apache Beam, Apache Samza, Apache Kafka, Apache Spark, and Apache Storm. One can easily extend Scotty with user-defined aggregation functions and window types. Scotty implements the concept of general stream slicing and derives workload characteristics from aggregation queries to improve performance without sacrificing its general applicability. We provide an in-depth view on the algorithms of the general stream slicing approach. Our experiments show that Scotty outperforms alternative solutions.

From natural language processing to neural databases

In recent years, neural networks have shown impressive performance gains on long-standing AI problems, such as answering queries from text and machine translation. These advances raise the question of whether neural nets can be used at the core of query processing to derive answers from facts, even when the facts are expressed in natural language. If so, it is conceivable that we could relax the fundamental assumption of database management, namely, that our data is represented as fields of a pre-defined schema. Furthermore, such technology would enable combining information from text, images, and structured data seamlessly. This paper introduces neural databases , a class of systems that use NLP transformers as localized answer derivation engines. We ground the vision in NeuralDB, a system for querying facts represented as short natural language sentences. We demonstrate that recent natural language processing models, specifically transformers, can answer select-project-join queries if they are given a set of relevant facts. However, they cannot scale to non-trivial databases nor answer set-based and aggregation queries. Based on these insights, we identify specific research challenges that are needed to build neural databases. Some of the challenges require drawing upon the rich literature in data management, and others pose new research opportunities to the NLP community. Finally, we show that with preliminary solutions, NeuralDB can already answer queries over thousands of sentences with very high accuracy.

WITHDRAWN: Compression techniques for performance optimisation in columnar databases

Data has become an integral part of today’s life. Traditional relational databases are row oriented databases designed for row wise data storage and retrieval which consumes more time to return queries. Recently, Columnar databases have dominated traditional databases due to their effective read operations to and from hard disk to decrease the interval to return queries. Columnar databases are mainly designed for data warehouses where huge amount of data collected from several sources are needed to perform Business Intelligence to build decision making systems. Columnar databases have quicker query performance because the data is packed close together which decreases seek time. They support various compression techniques for quicker data access with the help of aggregation queries to filter the irrelevant data. This paper discusses about various compression techniques such as NULL Suppression, Dictionary Encoding, Run Length Encoding, Bit Vector Encoding and Lempel Ziv Encoding which are being popularly used to optimise the performance of the columnar databases. Database operations are performed on the compressed data to prove the reduction in the requirements of memory and enhancements in the performance.

Structure-Free General Data Aggregation Scheduling for Multihop Battery-Free Wireless Networks

With advances in wireless power transfer techniques, battery-free wireless sensor networks (BF-WSNs) which can support long-term applications, has been attracting increasing interests in recent years. Unfortunately, the problem of minimum latency aggregation scheduling (MLAS) is not well studied in BF-WSNs. Existing works always have a rigid assumption that there is only one single query which is targeted at the whole network. Aiming at making the work more practical and general, we investigate the general MLAS problem in BF-WSNs, which is targeted at any subset of nodes in the network and aimed for an arbitrary number of aggregation queries. First, the general MLAS problem when there is one single query is studied. To control the number of nodes participating in the aggregation process, a node selection algorithm is proposed to cover and connect the whole target nodes. Then, a latency and energy aware scheduling algorithm is proposed to integrate the construction of aggregation tree with the chosen nodes, and the computation of a conflict-free schedule simultaneously, relying on non-predetermined structures. Second, the general MLAS problem when there is a group of aggregation queries is studied. Through designing some special structures to avoid collisions between both current and existing aggregation schedules, an algorithm without any waiting time is proposed. Additionally, the algorithm under physical interference model and dynamic energy arrival model are also presented. The theoretical analysis and simulation results verify that the proposed algorithms have high performance in terms of latency and energy efficiency.