Query Cost Research Articles

A new set of pivot elimination schemes for increasing the query performance

M-Tree, Slim-Tree, DF-Tree, and Omni-Tree are some of the popular dynamic structures which can grow incrementally by splitting overflowed nodes, and adding new levels to the tree very much like the B-tree variants. Unfortunately, they have been shown to perform very poorly compared to flat structur es such as AESA, LAESA, Spaghettis, and Kvp that use a fixed set of global pivots. HKvp index structure is an extension of Kvp allowing the elimination of pivots as well as the database objects. The number of pivots can be easily increased to provide more selectivity and query performance. However, there is an optimum number of pivots for a given query radius, and using too many pivots increases the costs of queries and index initialization. In this paper, a new set of pivot elimination mechanisms is proposed to determine the right number of pivots for different query radii. The suggested pivot elimination schemes perform significant cost reduction in terms of number of distance computations, and they estimate the drop rate value for HKvp on query time.

Accelerating SPARQL queries by exploiting hash-based locality and adaptive partitioning

State-of-the-art distributed RDF systems partition data across multiple computer nodes (workers). Some systems perform cheap hash partitioning, which may result in expensive query evaluation. Others try to minimize inter-node communication, which requires an expensive data preprocessing phase, leading to a high startup cost. Apriori knowledge of the query workload has also been used to create partitions, which, however, are static and do not adapt to workload changes. In this paper, we propose AdPart, a distributed RDF system, which addresses the shortcomings of previous work. First, AdPart applies lightweight partitioning on the initial data, which distributes triples by hashing on their subjects; this renders its startup overhead low. At the same time, the locality-aware query optimizer of AdPart takes full advantage of the partitioning to (1) support the fully parallel processing of join patterns on subjects and (2) minimize data communication for general queries by applying hash distribution of intermediate results instead of broadcasting, wherever possible. Second, AdPart monitors the data access patterns and dynamically redistributes and replicates the instances of the most frequent ones among workers. As a result, the communication cost for future queries is drastically reduced or even eliminated. To control replication, AdPart implements an eviction policy for the redistributed patterns. Our experiments with synthetic and real data verify that AdPart: (1) starts faster than all existing systems; (2) processes thousands of queries before other systems become online; and (3) gracefully adapts to the query load, being able to evaluate queries on billion-scale RDF data in subseconds.

Materialized Cube Selection Using Particle Swarm Optimization Algorithm

Storage of pre-computed views in data warehouse can essentially reduce query processing cost for decision support queries. The problem is to choose an optimal set of materialized views. Various frameworks such as lattice, MVPP and AND-OR graphs and algorithms like heuristic based, greedy, stochastic algorithm have been proposed in the literature for materialized view selection. Heuristic and greedy algorithms become slower in high dimensional search space while stochastic algorithms do not guarantee global optimal solution but reach to the optimum most solution in a fast and efficient way. In this paper we have implemented Particle Swarm Optimization (PSO) algorithm, one of the stochastic algorithm, on lattice framework to select an optimal set of views for materialization in data warehouse by minimizing query processing cost. We have compared our results with Genetic algorithm to prove the effectiveness of PSO algorithm over genetic algorithm.

Non-Volatile Nano-Electro-Mechanical Memory for Energy-Efficient Data Searching

A compact non-volatile nano-electro-mechanical memory (NV-NEMory) cell design together with a novel memory array architecture and operating scheme is proposed for real-time data searching applications. Performance characteristics of a vertically oriented NV-NEMory cell with a small layout area of $8F^{2}$ , where $F$ is the minimum half-pitch, are investigated by static and transient device simulations. Data searching can be achieved directly in the memory by a two-step read operation, dramatically improving the latency and energy cost of each search query.

On the Cost of Fixed Partial Match Queries in K-d Trees

Partial match queries constitute the most basic type of associative queries in multidimensional data structures such as $$K$$K-d trees or quadtrees. Given a query $$\mathbf {q}=(q_0,\ldots ,q_{K-1})$$q=(q0,?,qK-1) where s of the coordinates are specified and $$K-s$$K-s are left unspecified ($$q_i=*$$qi=?), a partial match search returns the subset of data points $$\mathbf {x}=(x_0,\ldots ,x_{K-1})$$x=(x0,?,xK-1) in the data structure that match the given query, that is, the data points such that $$x_i=q_i$$xi=qi whenever $$q_i\not =*$$qi??. There exists a wealth of results about the cost of partial match searches in many different multidimensional data structures, but most of these results deal with random queries. Only recently a few papers have begun to investigate the cost of partial match queries with a fixed query $$\mathbf {q}$$q. This paper represents a new contribution in this direction, giving a detailed asymptotic estimate of the expected cost $$P_{{n},\mathbf {q}}$$Pn,q for a given fixed query $$\mathbf {q}$$q. From previous results on the cost of partial matches with a fixed query and the ones presented here, a deeper understanding is emerging, uncovering the following functional shape for $$P_{{n},\mathbf {q}}$$Pn,q$$\begin{aligned} P_{{n},\mathbf {q}} = \nu \cdot \left( \prod _{i:q_i\text { is specified}}\, q_i(1-q_i)\right) ^{\alpha /2}\cdot n^\alpha + \text {l.o.t.} \end{aligned}$$Pn,q=?·?i:qiis specifiedqi(1-qi)?/2·n?+l.o.t.(l.o.t. lower order terms, throughout this work) in many multidimensional data structures, which differ only in the exponent $$\alpha $$? and the constant $$\nu $$?, both dependent on s and K, and, for some data structures, on the whole pattern of specified and unspecified coordinates in $$\mathbf {q}$$q as well. Although it is tempting to conjecture that this functional shape is universal, we have shown experimentally that it seems not to be true for a variant of $$K$$K-d trees called squarish $$K$$K-d trees.

Large-scale video copy retrieval with temporal-concentration SIFT

The scale-invariant feature transform (SIFT) feature plays a very important role in multimedia content analysis, such as near-duplicate image and video retrieval. However, the storage and query costs of SIFT become unbearable for large-scale databases. In this paper, SIFT features are robustly encoded with temporal information by tracking the SIFT to generate temporal-concentration SIFT (TCSIFT), which highly compresses the quantity of local features to reduce visual redundancy, and keeps the advantages of SIFT as much as possible at the same time. On the basis of TCSIFT, a novel framework for large-scale video copy retrieval is proposed in which the processes of retrieval and validation are implemented at the feature and frame level. Experimental results for two different datasets, i.e., CC_WEB_VIDEO and TRECVID, demonstrate that our method can yield comparable accuracy, compact storage size, and more efficient execution time, as well as adapt to various video transformations.

Development of technology for automated creation of materialized views

One of the urgent tasks in the DBMS optimization by the materialized views (MV) - the saved query evaluation results - is selecting queries, the result of which can be materialized. The problem is characterized by high computational complexity since it requires an analysis of the DBMS transaction log over a long period of time. The choice of the MV should be based on the cost of their future support, as well as the execution frequency and cost of queries, optimized by the MV data. It is effective to create the MP not for individual queries, but to form groups of similar queries and create views for them. When searching for the same type of queries, the SQL syntax should be taken into account. The paper considers the technology for automated creation of MV. Query evaluation was proposed, which takes into account the statistical and time indicators of query execution, as well as update frequency of the base tables. The query tokenization algorithm, which takes into account various syntactic structures of SQL was developed. The model of presenting queries in the form of numeric vectors was described. The pre-clustering stage was introduced to reduce the volume of the data processed. The improved clustering algorithm has allowed to form groups around the most resource-intensive and often executed queries. An algorithm for grouping and forming the central query, specifying the results of the preliminary clustering was developed. The described technology allows to automate the full creation cycle of MV. Experimental data showed that technology improvement has allowed to increase performance and reduce the resource intensity of the MV generation process, as well as enhance the quality of the developed MV compared to previous solutions.

Evaluating SPARQL queries on massive RDF datasets

Distributed RDF systems partition data across multiple computer nodes. Partitioning is typically based on heuristics that minimize inter-node communication and it is performed in an initial, data pre-processing phase. Therefore, the resulting partitions are static and do not adapt to changes in the query workload; as a result, existing systems are unable to consistently avoid communication for queries that are not favored by the initial data partitioning. Furthermore, for very large RDF knowledge bases, the partitioning phase becomes prohibitively expensive, leading to high startup costs. In this paper, we propose AdHash, a distributed RDF system which addresses the shortcomings of previous work. First, AdHash initially applies lightweight hash partitioning, which drastically minimizes the startup cost, while favoring the parallel processing of join patterns on subjects, without any data communication. Using a locality-aware planner, queries that cannot be processed in parallel are evaluated with minimal communication. Second, AdHash monitors the data access patterns and adapts dynamically to the query load by incrementally redistributing and replicating frequently accessed data. As a result, the communication cost for future queries is drastically reduced or even eliminated. Our experiments with synthetic and real data verify that AdHash (i) starts faster than all existing systems, (ii) processes thousands of queries before other systems become online, and (iii) gracefully adapts to the query load, being able to evaluate queries on billion-scale RDF data in sub-seconds. In this demonstration, audience can use a graphical interface of AdHash to verify its performance superiority compared to state-of-the-art distributed RDF systems.

OPTIMIZED INTENTION FOR CONTINUOUS QUERIES IN THE ACTIVE DATA AGGREGATION NETWORK WITH COST MODEL

Within an RDBMS streams of changes to the data and reporting when the result of a query defined over the data changes. These queries are referred to as the continuous queries since they continually produce results whenever new data arrives or existing data changes. To make online decision we require monitoring the continuous queries. Here the aim is to introduce the low-cost, scalable technique to answer continuous aggregation queries using a network of aggregators of dynamic data items. There is significant work in systems that can efficiently deliver the relevant updates automatically. And also to provide for getting the optimal set of sub queries with their incoherency bounds which satisfies client query’s coherency requirement with least number of refresh messages sent from aggregators to the client. For optimal query execution divide the query into sub-queries and evaluate each sub-query at a judiciously chosen data aggregator. The main purpose is to response the client with the least number of tasks with the help of random query selection. Random query selection means for the user submitted query the relevant queries, sub queries are generated. A query cost model which can be used to estimate the number of messages required to satisfy the client specified incoherency bound.

TraPlan: An Effective Three-in-One Trajectory-Prediction Model in Transportation Networks

The existing approaches for trajectory prediction (TP) are primarily concerned with discovering frequent trajectory patterns (FTPs) from historical movement data. Moreover, most of these approaches work by using a linear TP model to depict the positions of objects, which does not lend itself to the complexities of most real-world applications. In this research, we propose a three-in-one TP model in road-constrained transportation networks called TraPlan. TraPlan contains three essential techniques: 1) constrained network R-tree (CNR-tree), which is a two-tiered dynamic index structure of moving objects based on transportation networks; 2) a region-of-interest (RoI) discovery algorithm is employed to partition a large number of trajectory points into distinct clusters; and 3) a FTP-tree-based TP approach, called FTP-mining, is proposed to discover FTPs to infer future locations of objects moving within RoIs. In order to evaluate the results of the proposed CNR-tree index structure, we conducted experiments on synthetically generated data sets taken from real-world transportation networks. The results show that the CNR-tree can reduce the time cost of index maintenance by an average gap of about 40% when compared with the traditional NDTR-tree, as well as reduce the time cost of trajectory queries. Moreover, compared with fixed network R-Tree (FNR-trees), the accuracy of range queries has shown an on average improvement of about 32%. Furthermore, the experimental results show that the TraPlan demonstrates accurate and efficient prediction of possible motion curves of objects in distinct trajectory data sets by over 80% on average. Finally, we evaluate these results and the performance of the TraPlan model in regard to TP by comparing it with other TP algorithms.

Optimal nearest neighbor queries in sensor networks

Given a set of m mobile objects in a sensor network, we consider the problem of finding the nearest object among them from any node in the network at any time. These mobile objects are tracked by nearby sensors called proxy nodes. This problem requires an object tracking mechanism which typically relies on two basic operations: query and update. A query is invoked by a node each time when there is a need to find the closest object from it in the network. Updates of an object's location are initiated when the object moves from one location (proxy node) to another. We present a scalable distributed algorithm for tracking these mobile objects such that both the query cost and the update cost are small. The main idea in our algorithm is to maintain a virtual tree of downward paths pointing to the objects. Our algorithm guarantees an asymptotically optimal O(1) approximation for query cost and an O(min⁡{log⁡n,log⁡D}) approximation for update cost in the constant-doubling graph model, where n and D, respectively, are the number of nodes and the diameter of the network. We also give polylogarithmic approximations for both query and update cost in the general graph model. Our algorithm requires only polylogarithmic bits of memory per node. To the best of our knowledge, this is the first algorithm that is asymptotically optimal in handling nearest neighbor queries with low update cost in a distributed setting.

Optimal query assignment for wireless sensor networks

Increasing computing capabilities of modern sensors have enabled current wireless sensor networks to process queries within the network. This complements the traditional features of the sensor networks such as sensing the environment and communicating the data. Query processing, however, poses Quality of Service challenges such as query waiting time and validity (age) of the data. We focus on the processing cost of queries as a trade-off between the time queries wait to be processed and the age of the data provided to the queries. To model this trade-off, we propose a Continuous Time Markov Decision Process which assigns queries either to the sensor network, where queries wait to be processed, or to a central database, which provides stored and possibly outdated data. To compute an optimal query assignment policy, a Discrete Time Markov Decision Process, shown to be stochastically equivalent to the initial continuous time process, is formulated. A comparative numerical analysis of the performance of the optimal assignment policy and of several heuristics, derived from practice, is performed. This provides a theoretical support for the design and implementation of WSN applications, while ensuring a close-to-optimum performance of the system.

On hardness of several string indexing problems

Let D={d1,d2,…,dD} be a collection of D string documents of n characters in total. The two-pattern matching problems ask to index D for answering the following queries efficiently.•Report/count the unique documents containing P1 and P2.•Report/count the unique documents containing P1, but not P2.Here P1 and P2 represent input patterns of length p1 and p2 respectively. Linear space data structures with O(p1+p2+nklogO(1)⁡n) query cost are already known for the reporting version, where k represents the output size. For the counting version (i.e., report the value k), a simple linear-space index with O(p1+p2+n) query cost can be constructed in O(n3/2) time. However, it is still not known if these are the best possible bounds for these problems. In this paper, we show a strong connection between these string indexing problems and the boolean matrix multiplication problem. Based on this, we argue that these results cannot be improved significantly using purely combinatorial techniques. We also provide an improved upper bound for a related problem known as common colors query problem.

Walk, not wait

In this paper, we introduce a novel, general purpose, technique for faster sampling of nodes over an online social network. Specifically, unlike traditional random walks which wait for the convergence of sampling distribution to a predetermined target distribution - a waiting process that incurs a high query cost - we develop WALK-ESTIMATE, which starts with a much shorter random walk, and then proactively estimate the sampling probability for the node taken before using acceptance-rejection sampling to adjust the sampling probability to the predetermined target distribution. We present a novel backward random walk technique which provides provably unbiased estimations for the sampling probability, and demonstrate the superiority of WALK-ESTIMATE over traditional random walks through theoretical analysis and extensive experiments over real world online social networks.

Near-Optimal Location Tracking Using Sensor Networks

We consider the problem of tracking mobile objects using a sensor network. We present a distributed tracking algorithm, called Mobile Object Tracking using Sensors (MOT), that scales well with the number of sensors and also with the number of mobile objects. MOT maintains a hierarchical structure of detection lists of objects that can efficiently track mobile objects and resolve object queries at any time. MOT guarantees that the cost to update (or maintain) its data structures will be at most O (min { log n, log D } ) times the optimal update cost and query cost will be within O (1) of the optimal query cost in the constant-doubling graph model, where n and D , respectively, are the number of nodes and the diameter of the network. MOT achieves polylogarithmic approximations for both costs in the general graph model. MOT balances the object and bookkeeping information load at each node in the expense of only O (log n ) increase in the update and query costs. The experimentation evaluation in both one by one and concurrent execution situations shows that MOT performs well in practical scenarios. To the best of our knowledge, MOT is the first algorithm for this problem in a distributed setting that is traffic-oblivious, i.e. agnostic to a priori knowledge of objects movement patterns, mobility and query rate, etc., and is load balanced. All previous solutions for this problem assumed traffic-consciousness in constructing the tracking data structure.Â

HCBLS: A Hierarchical Cluster-Based Location Service in Urban Environment

Vehicle location information is central to many location-based services and applications in VANETs. Tracking vehicles positions and maintaining an accurate up-to-date view of the entire network are not easy due to the high mobility of vehicles and consequently rapid topology changes. The design of a scalable, accurate, and efficient location service is still a very challenging issue. In this paper, we propose a lightweight hierarchical cluster-based location service in city environments (HCBLS). HCBLS integrates a logical clustering based on the city digital map and consequently does not involve extra signaling overhead. An advanced location update aggregation at different levels of the assumed hierarchy is adopted to maintain up-to-date and accurate location information. Simulation results show that HCBLS achieves much better performances than the Efficient Map-Based Location Service (EMBLS) and any regular (non-cluster-based) updating scheme. HCBLS increases the success rate by around 10%, improves the overview of the network by more than 30%, lowers the location update and query costs by more than 7 times, lowers the message delivery latency by around 3 times, and presents around 4 times better localization accuracy.

Performance Evaluations of Graph Database using CUDA and OpenMP Compatible Libraries

Graph databases use graph structures to store data sets as nodes, edges, and properties. They are used to store and search the relationships between a large number of nodes, such as social networking services and recommendation engines that use customer social graphs. Since computation cost for graph search queries increases as the graph becomes large, in this pa- per we accelerate the graph search functions (Dijkstra and A* algorithms) of a graph database Neo4j using two ways: multi- threaded library and CUDA library for graphics processing units (GPUs). We use 100,000-node graphs generated based on a degree distribution of Facebook social graph for evaluations. Our multi-threaded and GPU-based implementations require an auxiliary adjacency matrix for a target graph. The results show that, when we do not take into account additional overhead to generate the auxiliary adjacency matrix, multi-threaded version improves the Dijkstra and A* search performance by 16.2x and 13.8x compared to the original implementation. The GPU-based implementation improves the Dijkstra and A* search performance by 26.2x and 32.8x. When we take into account the overhead, although the speed-ups by our implementations are reduced, by reusing the auxiliary adjacency matrix for multiple graph search queries we can significantly improve the graph search performance.

A bi-objective cost model for optimizing database queries in a multi-cloud environment

Abstract Cost models are broadly used in query processing to drive the query optimization process, accurately predict the query execution time, schedule database query tasks, apply admission control and derive resource requirements to name a few applications. The main role of cost models is to estimate the time needed to run the query on a specific machine. In a multi-cloud environment, cost models should be easily calibrated for a wide range of different physical machines, and time estimates need to be complemented with monetary cost information, since both the economic cost and the performance are of primary importance. This work aims to serve as the first proposal for a bi-objective query cost model suitable for queries executed over resources provided by potentially multiple cloud providers. We leverage existing calibrating modeling techniques for time estimates and we couple such estimates with monetary cost information covering the main charging options for using cloud resources. Moreover, we explain how the cost model can become part of an optimizer. Our approach is applicable to more generic data flow graphs, the execution plans of which do not necessarily comprise relational operators. Finally, we give a concrete example about the usage of our proposal and we validate its accuracy through real case studies.

A coding template for handling static and incremental horizontal partitioning in data warehouses

Today, data feeding warehouses come from devices with high speed and may affect the selected optimisation techniques such as horizontal data partitioning (HDP). HDP helps reduce the cost of complex OLAP queries and facilitates warehouse manageability. Selecting a partitioning schema for a given data warehouse is an Non-deterministic Polynomial-time hard (NP-hard) problem. Several studies exist which propose heuristics to select near-optimal solutions. Most of these heuristics are static, because they assume the existence of an a priori known set of queries. However, in real life, applications queries may change dynamically and the partitioning heuristics need to integrate these changes. An easy and naïve way to deal with this problem is to make the heuristics incremental by making efforts in implementation and coding levels. In this paper, we propose a new vision for handling the incremental selection of an HDP in a relational data warehouse. Our main efforts are concentrated on having a template coding that supports efficiently the changes of queries. First, we present a rich state of the art regarding the concepts of HDP (types, modes, algorithms). Second, we present a flexible coding to manage the static partitioning, where a genetic algorithm is proposed. Third, we extend our genetic algorithm to deal with the incremental aspects of queries. Finally, we conduct intensive experiments to validate our proposal.

GMOBench: Benchmarking generic moving objects

In real world scenarios, people's movement include several environments rather than one, for example, road network, pavement areas and indoor. This imposes a new challenge for moving objects database that the complete trip needs to be managed by a database system. In the meantime, novel queries regarding different transportation modes should also be supported. Since existing methods are limited to trips in a single environment and do not support queries on moving objects with different transportation modes, new technologies are essentially needed in a database system. In this paper, we introduce a benchmark called GMOBench that aims to evaluate the performance of a database system managing moving objects in different environments. GMOBench is settled in a realistic scenario and is comprised of three components: (1) a data generator with the capability of creating a scalable set of trips representing the complete movement of humans (both indoor and outdoor); (2) a set of carefully designed and benchmark queries; (3) Mode-RTree, an index structure for managing generic moving objects. The generator defines some parameters so that users can control the characteristics of results. We create the benchmark data in such a way that the dataset can mirror important characteristics and real world distributions of human mobility. Efficient access methods and optimization techniques are developed for query processing. In particular, we propose an index structure called Mode-RTree to manage moving objects in different environments. By employing the proposed index, the cost of benchmark queries is greatly reduced. GMOBench is implemented in a real database system to have a practical result. We perform an extensive experimental study on comprehensive datasets to evaluate the performance. The results show that by using the Mode-RTree we achieve significant performance improvement over the baseline method, demonstrating the effectiveness and efficiency of our approaches.