Aggregate Queries Research Articles

Towards Interactive Analytics over RDF Graphs

The continuous accumulation of multi-dimensional data and the development of Semantic Web and Linked Data published in the Resource Description Framework (RDF) bring new requirements for data analytics tools. Such tools should take into account the special features of RDF graphs, exploit the semantics of RDF and support flexible aggregate queries. In this paper, we present an approach for applying analytics to RDF data based on a high-level functional query language, called HIFUN. According to that language, each analytical query is considered to be a well-formed expression of a functional algebra and its definition is independent of the nature and structure of the data. In this paper, we investigate how HIFUN can be used for easing the formulation of analytic queries over RDF data. We detail the applicability of HIFUN over RDF, as well as the transformations of data that may be required, we introduce the translation rules of HIFUN queries to SPARQL and we describe a first implementation of the proposed model.

Wavelet-Based Least Common Ancestor Algorithm for Aggregate Query Processing in Energy Aware Wireless Sensor Network

Wireless sensor network (WSN) is developed as a network of sensors, which engage in sensing and transmitting the data to the sink node. The constraints, such as energy, memory, and bandwidth insist the researchers to develop an efficient method for data transmission in WSN. Accordingly, this paper introduces a data aggregation mechanism based on query processing, Wavelet-based Least Common Ancestor-Sliding window (WLCA-SW). The energy-loss and memory-crisis is well addressed using the proposed WLCA-SW through the successive steps of query processing, duplicate detection, data compression using the wavelet transformation, and data aggregation. The proposed WLCA-SWA is developed with the integration of the weighed sliding window and Least Common Ancestor (LCA), which enables the energy-aware aggregate query processing and de-duplication such that the duplicate records are detected potentially prior to the communication of the sensed data to the sink node. It is prominent that the weighed sliding window is the extension of the existing time-based sliding windows. The effectiveness of the proposed aggregate processing approach is evaluated based on the metrics, such as number of alive nodes, data reduction rate, data-loss percentage, and residual energy, which is found to be 33, 85%, 8.222%, and 0.0610 J at the end of 1000 rounds using 150 nodes for analysis. Moreover, the proposed method has the minimum aggregation error of 0.03, when the analysis is performed using 50 nodes.

SEQUOIA: A Middleware Supporting Policy-Based Access Control for Search and Aggregation in Data-Driven Applications

Application-level access control is a top priority when hardening software applications. In particular, run-time customization of access control policies and separation for concerns are becoming increasingly important. While these requirements are generally well-supported for request-response applications, there is a lack of support for data-focused operations, such as search or data aggregation, in a multi-tier architecture. Moreover, an ability to specify fine-grained access control policies is generally lacking for such applications. This puts at risk the security of organizations that employ existing and emerging database technologies and requires solutions that alleviate this issue. This paper approaches this issue through query rewriting. We present Sequoia , a data access middleware that enables attribute-based, application-level access control in data-driven applications. The middleware enforces external access control policies on data-focused operations such as search and aggregation queries by means of query rewriting based on dynamic run-time conditions. Sequoia provides run-time enforcement of policies that is scalable with regard to the database size. This paper presents an extensible architecture for both relational databases and document stores. It discusses the rewriting approach, and provides a formal verification of equivalencyand an extensive evaluation that shows that this approach scales better than the current state of practice and is an important track for future research.

The Cost of Privacy in Asynchronous Differentially-Private Machine Learning

We consider training machine learning models using data located on multiple private and geographically-scattered servers with different privacy settings. Due to the distributed nature of the data, communicating with all collaborating private data owners simultaneously may prove challenging or altogether impossible. We consider differentially-private asynchronous algorithms for collaboratively training machine-learning models on multiple private datasets. The asynchronous nature of the algorithms implies that a central learner interacts with the private data owners one-on-one whenever they are available for communication without needing to aggregate query responses to construct gradients of the entire fitness function. Therefore, the algorithm efficiently scales to many data owners. We define the cost of privacy as the difference between the fitness of a privacy-preserving machine-learning model and the fitness of trained machine-learning model in the absence of privacy concerns. We demonstrate that the cost of privacy has an upper bound that is inversely proportional to the combined size of the training datasets squared and the sum of the privacy budgets squared. We validate the theoretical results with experiments on financial and medical datasets. The experiments illustrate that collaboration among more than 10 data owners with at least 10,000 records with privacy budgets greater than or equal to 1 results in a superior machine-learning model in comparison to a model trained in isolation on only one of the datasets, illustrating the value of collaboration and the cost of the privacy. The number of the collaborating datasets can be lowered if the privacy budget is higher.

Video Monitoring Queries

Recent advances in video processing utilizing deep learning primitives achieved breakthroughs in fundamental problems in video analysis such as frame classification and object detection enabling an array of new applications. In this paper we study the problem of interactive declarative query processing on video streams. In particular we introduce a set of approximate filters to speed up queries that involve objects of specific type (e.g., cars, trucks, etc.) on video frames with associated spatial relationships among them (e.g., car left of truck). The resulting filters are able to assess quickly if the query predicates are true to proceed with further analysis of the frame or otherwise not consider the frame further avoiding costly object detection operations. We propose two classes of filters <inline-formula><tex-math notation="LaTeX">$IC$</tex-math></inline-formula> and <inline-formula><tex-math notation="LaTeX">$OD$</tex-math></inline-formula> , that adapt principles from deep image classification and object detection. The filters utilize extensible deep neural architectures and are easy to deploy and utilize. In addition, we propose statistical query processing techniques to process aggregate queries involving objects with spatial constraints on video streams and demonstrate experimentally the resulting increased accuracy on the resulting aggregate estimation. Finally, we introduce a framework based on extreme value theory to detect unexpected objects on video streams and experimentally demonstrate its utility. Combined these techniques constitute a robust set of video monitoring query processing techniques. We demonstrate that the application of the techniques proposed in conjunction with declarative queries on video streams can dramatically increase the frame processing rate and speed up query processing by at least two orders of magnitude. We present the results of a thorough experimental study utilizing benchmark video data sets at scale demonstrating the performance benefits and the practical relevance of our proposals.

Integrity Checking for Aggregate Queries

With the advent of cloud computing and Internet of Things and delegation of data collection and aggregation to third parties, the results of the computations should be verified. In distributed models, there are multiple sources. Each source creates authenticators for the values and sends them to the aggregator. The aggregator combines the authenticated values and creates a verification object for verifying the computation/aggregation results. In this paper, we propose two constructions for verifying the results of countable and window-based countable functions. These constructions are useful for aggregate functions such as median, max/min, top-k/first-k, and range queries, where the distribution of values is not visible for sources but is visible to the aggregator. The proposed constructions are secure based on the RSA problem in the random oracle model and have the correctness and succinctness properties. Experimental results show that the communication and computation costs of the constructions are acceptable in practice and the proposed solution can be employed for real-world applications.

Efficient Approximate Range Aggregation over Large-scale Spatial Data Federation

Range aggregation is a primitive operation in spatial data applications and there is a growing demand to support such operations over a data federation, where the entire spatial data are separately held by multiple data providers (a.k.a., data silos). Data federations notably increase the amount of data available for data-intensive applications such as smart mobility planning and public health emergency responses. Yet they also challenge the conventional implementation of range aggregation queries because the raw data cannot be shared within the federation and the data partition at each data silo is fixed during query processing. These constraints limit the design space of distributed range aggregation query processing. In this work, we propose approximate algorithms for efficient range aggregation over spatial data federation. We devise novel single-silo sampling algorithms that process queries in parallel and design a level sampling based algorithm which reduces the time complexity of local queries at each data silo to O(log 1/), where is the approximation ratio of the accuracy guarantee. Extensive evaluations with real-world data show that compared with state-of-the-arts, our solutions reduce the time cost and communication cost by up to 85.1x and 5.5x respectively, with average approximate errors of below 2.8%.

Functional Aggregate Queries with Additive Inequalities

Motivated by fundamental applications in databases and relational machine learning, we formulate and study the problem of answering functional aggregate queries (FAQ) in which some of the input factors are defined by a collection of additive inequalities between variables. We refer to these queries as FAQ-AI for short. To answer FAQ-AI in the Boolean semiring, we define relaxed tree decompositions and relaxed submodular and fractional hypertree width parameters. We show that an extension of the InsideOut algorithm using Chazelle’s geometric data structure for solving the semigroup range search problem can answer Boolean FAQ-AI in time given by these new width parameters. This new algorithm achieves lower complexity than known solutions for FAQ-AI. It also recovers some known results in database query answering. Our second contribution is a relaxation of the set of polymatroids that gives rise to the counting version of the submodular width, denoted by #subw. This new width is sandwiched between the submodular and the fractional hypertree widths. Any FAQ and FAQ-AI over one semiring can be answered in time proportional to #subw and respectively to the relaxed version of #subw. We present three applications of our FAQ-AI framework to relational machine learning: k -means clustering, training linear support vector machines, and training models using non-polynomial loss. These optimization problems can be solved over a database asymptotically faster than computing the join of the database relations.

CryptDICE: Distributed data protection system for secure cloud data storage and computation

Cloud storage allows organizations to store data at remote sites of service providers. Although cloud storage services offer numerous benefits, they also involve new risks and challenges with respect to data security and privacy aspects. To preserve confidentiality, data must be encrypted before outsourcing to the cloud. Although this approach protects the security and privacy aspects of data, it also impedes regular functionality such as executing queries and performing analytical computations. To address this concern, specific data encryption schemes (e.g., deterministic, random, homomorphic, order-preserving, etc.) can be adopted that still support the execution of different types of queries (e.g., equality search, full-text search, etc.) over encrypted data.However, these specialized data encryption schemes have to be implemented and integrated in the application and their adoption introduces an extra layer of complexity in the application code. Moreover, as these schemes imply trade-offs between performance and security, storage efficiency, etc, making the appropriate trade-off is a challenging and non-trivial task. In addition, to support aggregate queries, User Defined Functions (UDF) have to be implemented directly in the database engine and these implementations are specific to each underlying data storage technology, which demands expert knowledge and in turn increases management complexity.In this paper, we introduce CryptDICE, a distributed data protection system that (i) provides built-in support for a number of different data encryption schemes, made accessible via annotations that represent application-specific (search) requirements; (ii) supports making appropriate trade-offs and execution of these encryption decisions at diverse levels of data granularity; and (iii) integrates a lightweight service that performs dynamic deployment of User Defined Functions (UDF) –without performing any alteration directly in the database engine– for heterogeneous NoSQL databases in order to realize low-latency aggregate queries and also to avoid expensive data shuffling (from the cloud to an on-premise data center). We have validated CryptDICE in the context of a realistic industrial SaaS application and carried out an extensive functional validation, which shows the applicability of the middleware platform. In addition, our experimental evaluation efforts confirm that the performance overhead of CryptDICE is acceptable and validates the performance optimizations for achieving low-latency aggregate queries.

A Resilient Large-Scale Trajectory Index for Cloud-Based Moving Object Applications

The availability of location-aware devices generates tremendous volumes of moving object trajectories. The processing of these large-scale trajectories requires innovative techniques that are capable of adapting to changes in cloud systems to satisfy a wide range of applications and non-programmer end users. We introduce a Resilient Moving Object Index that is capable of balancing both spatial and object localities to maximize the overall performance in numerous environments. It is equipped with compulsory, discrete, and impact factor prediction models. The compulsory and discrete models are used to predict a locality pivot based on three fundamental aspects: computation resources, nature of the trajectories, and query types. The impact factor model is used to predict the influence of contrasting queries. Moreover, we provide a framework to extract efficient training sets and features without adding overhead to the index construction. We conduct an extensive experimental study to evaluate our approach. The evaluation includes two testbeds and covers spatial, temporal, spatio-temporal, continuous, aggregation, and retrieval queries. In most cases, the experiments show a significant performance improvement compared to various indexing schemes on a compact trajectory dataset as well as a sparse dataset. Most important, they demonstrate how our proposed index adapts to change in various environments.

A general framework for privacy-preserving of data publication based on randomized response techniques

Privacy preserving is a paramount concern in publishing datasets that contain sensitive information. Preventing privacy disclosure and providing useful information to legitimate users for data analyzing/mining are conflicting goals. Randomized response is a class of techniques that perturbs each sensitive value in a certain way, so that personal privacy is protected while the large-trend of the entire dataset is still recoverable. However, existing randomized response techniques do not allow to flexibly configure the level of privacy protection, support only a few types of aggregate queries, and cannot achieve the best answer accuracy from perturbed data. These drawbacks impair the effectiveness of those techniques. This paper proposes a general framework based on randomized response techniques, which has good flexibility and extensibility, and can improve the effectiveness of randomized response methods. Our approach is validated by extensive experiments and comparison with existing randomized response and generalization methods.

Large-scale Data Exploration Using Explanatory Regression Functions

Analysts wishing to explore multivariate data spaces, typically issue queries involving selection operators, i.e., range or equality predicates, which define data subspaces of potential interest. Then, they use aggregation functions, the results of which determine a subspace’s interestingness for further exploration and deeper analysis. However, Aggregate Query (AQ) results are scalars and convey limited information and explainability about the queried subspaces for enhanced exploratory analysis. Analysts have no way of identifying how these results are derived or how they change w.r.t query (input) parameter values. We address this shortcoming by aiding analysts to explore and understand data subspaces by contributing a novel explanation mechanism based on machine learning. We explain AQ results using functions obtained by a three-fold joint optimization problem which assume the form of explainable piecewise-linear regression functions. A key feature of the proposed solution is that the explanation functions are estimated using past executed queries. These queries provide a coarse grained overview of the underlying aggregate function (generating the AQ results) to be learned. Explanations for future, previously unseen AQs can be computed without accessing the underlying data and can be used to further explore the queried data subspaces, without issuing more queries to the backend analytics engine. We evaluate the explanation accuracy and efficiency through theoretically grounded metrics over real-world and synthetic datasets and query workloads.

Graph cells: Top-k structural-textual aggregated query over information networks

The graph OLAP aggregated analysis in information networks has been extensively studied. However, previous works have neglected to integrate the structural information into this kind of query and ignored the influence of enough textual information in graph aggregation operations. In this paper, we propose a novel OLAP query called top-k structural-textual aggregated graph cell query to analyze the information data. According to the given keywords, this query is to find top-k structural-textual aggregated graph cells in text-rich multidimensional information networks. Under the conditions of matching attribution values in a portion of dimensions, a graph cell is defined as a subgraph of the network. It only contains documents of all included vertices in this subgraph. To distinguish the importance of different graph cells, we firstly design a dominating number-based threshold testing and a flexible ranking function integrating the text similarity with the query and the structural size to obtain k most relevant graph cells. Then, we propose a new hybrid index structure and a filtering-and-verification framework, which includes an efficient search algorithm and several pruning and bounding techniques. Finally, we verify the effectiveness and efficiency of the proposed methods through extensive experiments.

Demonstration of ScroogeDB

We demonstrate ScroogeDB which aims at minimizing monetary cost of processing aggregation queries in the Cloud. It runs on top of a Cloud database that offers pay-as-you-go query processing where users pay according to the number of bytes processed. ScroogeDB exploits deterministic approximate query processing (DAQ) to achieve monetary savings. That is, ScroogeDB provides deterministic bounds, i.e., bounds that contain the true value with a 100% probability. ScroogeDB creates small synopses of the database and uses these synopses to answer aggregation queries. By rewriting a query on base tables into a query on smaller synopses, we significantly reduce the amount of processed data. We do not pre-compute synopses in advance of an analysis session. Instead, we generate them on the fly, interleaving synopsis generation with query execution. In our demonstration, we show that our system realizes impressive monetary savings with little precision loss. We run our system on top of the Google BigQuery Cloud platform and provide users with a graphical interface that visualizes deterministic bounds. The graphical interface also provides information regarding the generated synopses and how they contribute to monetary savings.

Processing Location-Based Aggregate Queries in Road Networks

In recent years, the research community has introduced various methods for processing the location-based queries on a single type of objects in road networks. However, in real-life applications user may be interested in obtaining information about different types of objects, in terms of their neighboring relationship. The sets of different types of objects closer to each other are termed the heterogeneous neighboring object sets (or HNO sets for short). To provide users with object information by considering both the spatial closeness of objects to the query object and the neighboring relationship between objects, we present useful and important location-based aggregate queries on finding the HNO sets in road net-works. The location-based aggregate queries are the shortest average distance query (SAvgDQ), the shortest minimal distance query (SMinDQ), the shortest maximal distance query (SMaxDQ), and the shortest sum distance query (SSumDQ). We first utilize a grid index to manage information of data objects and road networks, and then propose the SAvgDQ, SMinDQ, SMaxDQ, and SSumDQ processing algorithms, which are combined with the grid index to efficiently process the four types of location-based aggregate queries, respectively. A comprehensive set of experiments is conducted to demonstrate the efficiency of the proposed processing algorithms using real road network datasets.

Chanjo: Clincal grade sequence coverage analysis

Coverage analysis is essential when analysing massive parallel sequencing (MPS) data. The analysis indicates existence of false negatives or positives in a region of interest or poorly covered genomic regions. There are several tools that have excellent performance when doing coverage analysis on a few samples with predefined regions. However, there is no current tool for collecting samples over a longer period of time for aggregated coverage analysis of multiple samples or sequencing methods. Furthermore, current coverage analysis tools do not generate customized coverage reports or enable exploratory coverage analysis without extensive bioinformatic skill and access to the original alignment files. We present Chanjo, a user friendly coverage analysis tool for persistent storage of coverage data, that, accompanied with Chanjo Report, produces coverage reports that summarize coverage data for predefined regions in an elegant manner. Chanjo Report can produce both structured coverage reports and dynamic reports tailored to a subset of genomic regions, coverage cut-offs or samples. Chanjo stores data in an SQL database where thousands of samples can be added over time, which allows for aggregate queries to discover problematic regions. Chanjo is well tested, supports whole exome and genome sequencing, and follows common UNIX standards, allowing for easy integration into existing pipelines. Chanjo is easy to install and operate, and provides a solution for persistent coverage analysis and clinical-grade reporting. It makes it easy to set up a local database and automate the addition of multiple samples and report generation. To our knowledge there is no other tool with matching capabilities. Chanjo handles the common file formats in genetics, such as BED and BAM, and makes it easy to produce PDF coverage reports that are highly valuable for individuals with limited bioinformatic expertise. We believe Chanjo to be a vital tool for clinicians and researchers performing MPS analysis.

Multi-dimensional aggregation: a viable solution for interval data

Now a day multi-dimensional data modeling and aggregate query processing which are key assets of business intelligence solutions are being frequently realized to the unorthodox data. For interval values which are recorded when the data is on hold, multidimensional aggregation is the only viable solution and the author emphasizes over this aspect in this paper. Actually, such intervals reflect the state of reality of either current data or such data which were part of the present database. Every possible challenge which interval data throws upon is resolved in this paper through introduction of aggregation operator. Although the intervals are unknown at first but they eventually depend on the actual data and it turns out to be quiet handy while associating them with the resulting tuples. Only those result groups are selected for this purpose, which are specified partially. The interval data signifies that data holds either for each interim in the interval or entire interval and in both of these two cases it faces contention with the operators. In this paper, the author presents the empirical analysis of the aggregation operator after its implementation over the huge industrial data sets and claims that it holds an edge over the other temporal aggregation algorithms.

Advancing clinical cohort selection with genomics analysis on a distributed platform.

The affordability of next-generation genomic sequencing and the improvement of medical data management have contributed largely to the evolution of biological analysis from both a clinical and research perspective. Precision medicine is a response to these advancements that places individuals into better-defined subsets based on shared clinical and genetic features. The identification of personalized diagnosis and treatment options is dependent on the ability to draw insights from large-scale, multi-modal analysis of biomedical datasets. Driven by a real use case, we premise that platforms that support precision medicine analysis should maintain data in their optimal data stores, should support distributed storage and query mechanisms, and should scale as more samples are added to the system. We extended a genomics-based columnar data store, GenomicsDB, for ease of use within a distributed analytics platform for clinical and genomic data integration, known as the ODA framework. The framework supports interaction from an i2b2 plugin as well as a notebook environment. We show that the ODA framework exhibits worst-case linear scaling for array size (storage), import time (data construction), and query time for an increasing number of samples. We go on to show worst-case linear time for both import of clinical data and aggregate query execution time within a distributed environment. This work highlights the integration of a distributed genomic database with a distributed compute environment to support scalable and efficient precision medicine queries from a HIPAA-compliant, cohort system in a real-world setting. The ODA framework is currently deployed in production to support precision medicine exploration and analysis from clinicians and researchers at UCLA David Geffen School of Medicine.

Adaptive learning of aggregate analytics under dynamic workloads

Large organizations have seamlessly incorporated data-driven decision making in their operations. However, as data volumes increase, expensive big data infrastructures are called to rescue. In this setting, analytics tasks become very costly in terms of query response time, resource consumption, and money in cloud deployments, especially when base data are stored across geographically distributed data centers. Therefore, we introduce an adaptive, reciprocity-based Machine Learning mechanism which is light-weight, stored client-side, can estimate the answers of a variety of aggregate queries and can avoid the big data back-end. The estimations are performed in milliseconds are inexpensive and accurate as the mechanism learns from past analytical-query patterns. However, as analytic queries are ad hoc and analysts’ interests change over time we develop solutions that can swiftly and accurately detect such changes and adapt to new query patterns. The capabilities of our approach are demonstrated using extensive evaluation with real and synthetic datasets.

Obscure: Information-Theoretically Secure, Oblivious, and Verifiable Aggregation Queries on Secret-Shared Outsourced Data

Despite exciting progress on cryptography, secure and efficient query processing over outsourced data remains an open challenge. We develop a communication-efficient and information-theoretically secure system, entitled Obscure for aggregation queries with conjunctive or disjunctive predicates, using secret-sharing. Obscure is strongly secure (i.e., secure regardless of the computational-capabilities of an adversary) and prevents the network, as well as, the (adversarial) servers to learn the user's queries, results, or the database. In addition, Obscure provides additional security features, such as hiding access-patterns (i.e., hiding the identity of the tuple satisfying a query) and hiding query-patterns (i.e., hiding which two queries are identical). Also, Obscure does not require any communication between any two servers that store the secret-shared data before/during/after the query execution. Moreover, our techniques deal with the secret-shared data that is outsourced by a single or multiple database owners, as well as, allows a user, which may not be the database owner, to execute the query over secret-shared data. We further develop (non-mandatory) privacy-preserving result verification algorithms that detect malicious behaviors, and experimentally validate the efficiency of Obscure on large datasets, the size of which prior approaches of secret-sharing or multi-party computation systems have not scaled to.