Optimize Query Processing Research Articles

Optimization of Query Processing in Versatile Database Using Ant Colony Algorithm

Optimization of Query Processing in Versatile Database Using Ant Colony Algorithm

Innovations in Cyber Security Algorithms for Databases Enhancing Data Retrieval and Management

The term Innovations in Cyber Security Algorithms for Databases Enhancing Data Retrieval and Management refers to a book that studies novel techniques for tackling problems related to digital data. The integration of three complicated methods—DQO, DSS, and RAI—is the major focus of attention in this piece of writing. DQO makes use of machine learning to optimize query processing on the fly to meet fluctuating workloads. This is done to accommodate such workloads. To address issues pertaining to the scale of distributed systems, distributed storage systems (DSS) convey data in an effective manner by utilizing consistent hashing. The RAI algorithm adjusts the index architecture in response to the query patterns to achieve real-time flexibility. In this way, the process of looking for information that is frequently asked about is sped up. The methodology that has been suggested is superior to six different ways that are often used in terms of its adaptability, scalability, and real-time capabilities. This article will give a thorough model for improving data management in computer systems. The objective of this essay is to present the model.

Algorithms and methods for managing request flows in a distributed service system

The article devoted to the approaches to algorithmization of the information management system of a distributed service enterprise. The study of the dependence of the optimal capacity of the service system center on the service area was carried out. The mechanism of optimization of the service area, taking into account organizational and economic constraints, as well as optimization of the service area for mobile sources of applications, is investigated. Processing customer requests must meet the requirements of efficiency, reliability, and efficiency of the corporate information network. In the presence of a single office of a service company, it is not always possible to find an optimal or close to optimal query processing algorithm in terms of performance. The problem of the optimal service area without taking into account restrictions in order to maximize the profit of the enterprise is considered. Models of the functioning of the decision-making subsystem for servicing requests by a service enterprise have been developed, differing in taking into account the service area and the spatial stationarity of customers.

Simple Adaptive Query Processing vs. Learned Query Optimizers: Observations and Analysis

There have been many decades of work on optimizing query processing in database management systems. Recently, modern machine learning (ML), and specifically reinforcement learning (RL), has gained increased attention as a means to develop a query optimizer (QO). In this work, we take a closer look at two recent state-of-the-art (SOTA) RL-based QO methods to better understand their behavior. We find that these RL-based methods do not generalize as well as it seems at first glance. Thus, we ask a simple question: How do SOTA RL-based QOs compare to a simple, modern, adaptive query processing approach? To answer this question, we choose two simple adaptive query processing techniques and implemented them in PostgreSQL. The first adapts an individual join operation on-the-fly and switches between a Nested Loop Join algorithm and a Hash Join algorithm to avoid sub-optimal join algorithm decisions. The second is a technique called Lookahead Information Passing (LIP), in which adaptive semijoin techniques are used to make a pipeline of join operations execute efficiently. To our surprise, we find that this simple adaptive query processing approach is not only competitive to the SOTA RL-based approaches but, in some cases, outperforms the RL-based approaches. The adaptive approach is also appealing because it does not require an expensive training step, and it is fully interpretable compared to the RL-based QO approaches. Further, the adaptive method works across complex query constructs that RL-based QO methods currently cannot optimize.

Partition-based SIMD Processing and its Application to Columnar Database Systems

The Single Instruction Multiple Data (SIMD) paradigm became a core principle for optimizing query processing in columnar database systems. Until now, only the LOAD/STORE instructions are considered to be efficient enough to achieve the expected speedups, while avoiding GATHER/SCATTER is considered almost imperative. However, the GATHER instruction offers a very flexible way to populate SIMD registers with data elements coming from non-consecutive memory locations. As we will discuss within this article, the GATHER instruction can achieve the same performance as the LOAD instruction, if applied properly. To enable the proper usage, we outline a novel access pattern allowing fine-grained, partition-based SIMD implementations. Then, we apply this partition-based SIMD processing to two representative examples from columnar database systems to experimentally demonstrate the applicability and efficiency of our new access pattern.

Optimization of Query Processing on Multi-tiered Persistent Storage

Optimization of Query Processing on Multi-tiered Persistent Storage

Benchmarking JSON Document Stores in Practice

The increasing dissemination of JSON as exchange and storage format through its popularity in business and analytical applications requires efficient storage and processing of JSON documents. Consequently, this led to the development of specialized JSON document stores and the extension of existing relational stores, while no JSON-specific benchmarks were available to assess these systems. In this work, we assess currently available JSON document store benchmarks and select the recently developed DeepBench benchmark to experimentally study important dimensions like analytical querying capabilities, object nesting and array unnesting. To make the computational complexity of array unnesting more tractable, we introduce an improvement that we evaluate within a commercial system as part of the common, performance-oriented development process in practice. We conclude our evaluation of well-known document stores with DeepBench and give new insights into strengths and potential weaknesses of those systems that were not found by existing, non-JSON benchmarking practices. In particular the algebraic optimization of JSON query processing is still limited despite prior work on hierarchical data models in the XML context.

Search and analytics using semantic annotations

Search and analytics using semantic annotations

An Efficient Indexing Approach for Continuous Spatial Approximate Keyword Queries over Geo-Textual Streaming Data

Current social-network-based and location-based-service applications need to handle continuous spatial approximate keyword queries over geo-textual streaming data of high density. The continuous query is a well-known expensive operation. The optimization of continuous query processing is still an open issue. For geo-textual streaming data, the performance issue is more serious since both location information and textual description need to be matched for each incoming streaming data tuple. The state-of-the-art continuous spatial-keyword query indexing approaches generally lack both support for approximate keyword matching and high-performance processing for geo-textual streaming data. Aiming to tackle this problem, this paper first proposes an indexing approach for efficient supporting of continuous spatial approximate keyword queries by integrating m i n - w i s e signatures into an AP-tree, namely AP-tree + . AP-tree + utilizes the one-permutation m i n - w i s e hashing method to achieve a much lower signature maintenance costs compared with the traditional m i n - w i s e hashing method because it only employs one hashing function instead of dozens. Towards providing a more efficient indexing approach, this paper has explored the feasibility of parallelizing AP-tree + by employing a Graphic Processing Unit (GPU). We mapped the AP-tree + data structure into the GPU’s memory with a variety of one-dimensional arrays to form the GPU-aided AP-tree + . Furthermore, a m i n - w i s e parallel hashing algorithm with a scheme of data parallel and a GPU-CPU data communication method based on a four-stage pipeline way have been used to optimize the performance of the GPU-aided AP-tree + . The experimental results indicate that (1) AP-tree + can reduce the space cost by about 11% compared with MHR-tree, (2) AP-tree + can hold a comparable recall and 5.64× query performance gain compared with MHR-tree while saving 41.66% maintenance cost on average, (3) the GPU-aided AP-tree + can attain an average speedup of 5.76× compared to AP-tree + , and (4) the GPU-CPU data communication scheme can further improve the query performance of the GPU-aided AP-tree + by 39.4%.

Generalizing the Pigeonhole Principle for Similarity Search in Hamming Space

A distance search in Hamming space finds binary vectors whose Hamming distances are no more than a threshold from a query vector. It is a fundamental problem in many applications, such as image retrieval, near-duplicate Web page detection, and scientific databases. State-of-the-art approaches to Hamming distance search are mainly based on the pigeonhole principle to generate a set of candidates and then verify them. We observe that the constraint by the pigeonhole principle is not always tight and may bring about unnecessary candidates. We also observe that the distribution in real data is often skewed, but most existing solutions adopt a simple equi-width partitioning and allocate the same threshold to all the parts, hence failing to exploit the data skewness to optimize query processing. In this paper, we propose a new form of the pigeonhole principle which allows variable partitioning and threshold allocation. Based on the new principle, we develop a tight constraint of candidates and devise cost-aware methods for partitioning and threshold allocation to optimize query processing. In addition, we extend our methods to answer Hamming distance join queries. We also discuss the application of the pigeonhole principle in set similarity search, a problem that can be converted to Hamming distance search equivalently. Our evaluation on datasets with various data distributions shows the robustness of our solution and its superior query processing performance to the state-of-the-art methods.

The Optimization of Query Processing in Seabase Cloud Databases based on CCEVP Model

A cloud database is a database usually installed on cloud computing software platforms. There are several methods for query processing in cloud databases. This study tried to optimize query processing in the SeaBase cloud database and reduce query processing time. This method used adaptability for optimization. This method was designed for cloud-based databases. The algorithm is composed of three components: 1) multi cloud query separator; 2) query similarity detector based on the execution plan; and 3) replacement policy. This method is implemented as a system for a fully object-oriented simulation. The system is added to the SeaBase as an agent. The evaluation result show that this method reduced response time by 1.9 percent.

The Optimization of Query Processing in Sea Base Cloud Databases Based on CCEVP Model

A cloud database is a distributed database that can provide services for distributed data. The Sea Base is a cloud database that can offer integrated services from different databases. Several methods have been proposed for query processing in cloud databases. This study intends to propose a new method to optimize query processing in cloud databases. By detecting frequent queries sent to the database and keeping their execution plans, this method attempts to optimize query processing in the Sea Base. It is composed of three parts: separator, similarity detector and replacement policy. The results of system simulation show, that the system time response is reduced by 1.9 percent with this method. In future, this optimization can improve by improving the said parts.

Materialized View Selection using Artificial Bee Colony Optimization

Data warehouse is an essential component of almost every modern enterprise information system. It stores huge amount of subject-oriented, time-stamped, non-volatile and integrated data. It is highly required of the system to respond to complex online analytical queries posed against its data warehouse in seconds for efficient decision making. Optimization of online analytical query processing (OLAP) could substantially minimize delays in query response time. Materialized view is an efficient and effective OLAP query optimization technique to minimize query response time. Selecting a set of such appropriate views for materialization is referred to as view selection, which is a nontrivial task. In this regard, an Artificial Bee Colony (ABC) based view selection algorithm (ABCVSA), which has been adapted by incorporating N-point and GBFS based N-point random insertion operations, to select Top-K views from a multidimensional lattice is proposed. Experimental results show that ABCVSA performs better than the most fundamental view selection algorithm HRUA. Thus, the views selected using ABCVSA on materialization would reduce the query response time of OLAP queries and thereby aid analysts in arriving at strategic business decisions in an effective manner.

An evaluation of analytical queries on CPUs and coupled GPUs

SummaryRecently, the mainstream hardware vendors such as Intel and AMD have made significant efforts to integrate the central processing unit (CPU) and the graphics processing unit (GPU) into a single chip, which forms a coupled CPU‐GPU architecture. Data transfer between the CPU and the GPU through a Peripheral Component Interconnect Express bus is eliminated on this architecture, which provides new opportunities for database community to optimize query processing. Because of the lack of comprehensive evaluation of database systems on coupled CPU‐GPU platforms, it is difficult for academic and industry researchers to make appropriate decisions on improvement and optimization directions. In this paper, we conduct an extensive experimental study to evaluate an online analytical processing system on Intel and AMD machines. The performance difference is measured and analyzed when executing queries on integrated GPUs and multicore CPUs. The impacts of various parameters, data sizes, and optimization techniques on performance are also investigated. The results provide preliminary insights into database query and operator behaviors on state‐of‐the‐art coupled CPU‐GPU architectures.

Efficient Query Processing in 3D Motion Capture Gesture Databases

One of the most fundamental challenges when accessing gestural patterns in 3D motion capture databases is the definition of spatiotemporal similarity. While distance-based similarity models such as the Gesture Matching Distance on gesture signatures are able to leverage the spatial and temporal characteristics of gestural patterns, their applicability to large 3D motion capture databases is limited due to their high computational complexity. To this end, we present a lower bound approximation of the Gesture Matching Distance that can be utilized in an optimal multi-step query processing architecture in order to support efficient query processing. We investigate the performance in terms of accuracy and efficiency based on 3D motion capture databases and show that our approach is able to achieve an increase in efficiency of more than one order of magnitude with a negligible loss in accuracy. In addition, we discuss different applications in the digital humanities in order to highlight the significance of similarity search approaches in the research field of gestural pattern analysis.

Load-aware inter-co-processor parallelism in database query processing

For a decade, the database community has been exploring graphics processing units and other co-processors to accelerate query processing. While the developed algorithms often outperform their CPU counterparts, it is not beneficial to keep processing devices idle while overutilizing others. Therefore, an approach is needed that efficiently distributes a workload on available (co-)processors while providing accurate performance estimates for the query optimizer. In this paper, we contribute heuristics that optimize query processing for response time and throughput simultaneously via inter-device parallelism. Our empirical evaluation reveals that the new approach achieves speedups up to 1.85 compared to state-of-the-art approaches while preserving accurate performance estimations. In a further series of experiments, we evaluate our approach on two new use cases: joining and sorting. Furthermore, we use a simulation to assess the performance of our approach for systems with multiple co-processors and derive some general rules that impact performance in those systems.

A SURVEY ON OPTIMAL ROUTE QUERIES FOR ROAD NETWORKS

In daily life the need to find optimal routes betwe en two points is critical, for example finding the shortest distance to the nearest hospital. Internet based maps are now widely used f or this purpose. Route search and optimal route que ries are two important classes of queries based on road network concept. Route sea rch queries find the route according to the given c onstraints. The optimal route queries find the optimum route from a set of specif ications by a user. In road map queries, users hav e to give the specification of starting point and ending point of their travelling with or without constraints. Some spatial features about the categories and the different locations should be specified along with this. If the travelling constraints are given then it should be unique. These constraints may be either total order or partial or der. In this specification order there should be in formation about both starting point and destination point of the travelling. The optima l route queries optimize the possible routes and gi ve the optimal route that satisfies all the constraints. This paper describes the surve y on optimal route query processing, two categories namely optimal route query processing and spatial search with categorical info rmation have been considered, a discussion on techn ique for optimal route query with constraints and without constraint is also inc luded. The total order needs a specification of lis t of points and in the same order that they should be visited but that is not require d for partial order constraints. Finally this paper concludes with pros and cons of different techniques under optimal route queries.

The Yin and Yang of processing data warehousing queries on GPU devices

Database community has made significant research efforts to optimize query processing on GPUs in the past few years. However, we can hardly find that GPUs have been truly adopted in major warehousing production systems. Preparing to merge GPUs to the warehousing systems, we have identified and addressed several critical issues in a three-dimensional study of warehousing queries on GPUs by varying query characteristics, software techniques, and GPU hardware configurations. We also propose an analytical model to understand and predict the query performance on GPUs. Based on our study, we present our performance insights for warehousing query execution on GPUs. The objective of our work is to provide a comprehensive guidance for GPU architects, software system designers, and database practitioners to narrow the speed gap between the GPU kernel execution (the fast mode) and data transfer to prepare GPU execution (the slow mode) for high performance in processing data warehousing queries. The GPU query engine developed in this work is open source to the public.

Distributed Query Plan Generation using Particle Swarm Optimization

A large number of queries are posed on databases spread across the globe. In order to process these queries efficiently, optimal query processing strategies that generate efficient query processing plans are being devised. In distributed relational database systems, due to replication of relations at multiple sites, the relations required to answer a query may necessitate accessing of data from multiple sites. This leads to an exponential increase in the number of possible alternative query plans for processing a query. Though it is not computationally feasible to explore all possible query plans in such a large search space, the query plan that provides the most cost-effective option for query processing is considered necessary and should be generated for a given query. In this paper, an attempt has been made to generate such optimal query plans using Set based Comprehensive Learning Particle Swarm Optimization (S-CLPSO). Experimental comparisons of this algorithm with the GA based distributed query plan generation algorithm shows that for higher number of relations, the S-CLPSO based algorithm is able to generate comparatively better quality Top-K query plans.

$\mathcal{MD}$ -HBase: design and implementation of an elastic data infrastructure for cloud-scale location services

The ubiquity of location enabled devices has resulted in a wide proliferation of location based applications and services. To handle the growing scale, database management systems driving such location based services (LBS) must cope with high insert rates for location updates of millions of devices, while supporting efficient real-time analysis on latest location. Traditional DBMSs, equipped with multi-dimensional index structures, can efficiently handle spatio-temporal data. However, popular open-source relational database systems are overwhelmed by the high insertion rates, real-time querying requirements, and terabytes of data that these systems must handle. On the other hand, key-value stores can effectively support large scale operation, but do not natively provide multi-attribute accesses needed to support the rich querying functionality essential for the LBSs. We present the design and implementation of $\mathcal {MD}$ -HBase, a scalable data management infrastructure for LBSs that bridges this gap between scale and functionality. Our approach leverages a multi-dimensional index structure layered over a key-value store. The underlying key-value store allows the system to sustain high insert throughput and large data volumes, while ensuring fault-tolerance, and high availability. On the other hand, the index layer allows efficient multi-dimensional query processing. Our optimized query processing technique accesses only the index and storage level entries that intersect with the query region, thus ensuring efficient query processing. We present the design of $\mathcal {MD}$ -HBase that demonstrates how two standard index structures--the K-d tree and the Quad tree--can be layered over a range partitioned key-value store to provide scalable multi-dimensional data infrastructure. Our prototype implementation using HBase, a standard open-source key-value store, can handle hundreds of thousands of inserts per second using a modest 16 node cluster, while efficiently processing multi-dimensional range queries and nearest neighbor queries in real-time with response times as low as few hundreds of milliseconds.