Query Processing Research Articles

Bag Semantics Conjunctive Query Containment. Four Small Steps Towards Undecidability.

Query Containment Problem (QCP) is one of the most fundamental decision problems in database query processing and optimization. Complexity of QCP for conjunctive queries has been fully understood since 1970s. But, as Chaudhuri and Vardi noticed in their classical 1993 paper this understanding is based on the assumption that query answers are sets of tuples, and it does not transfer to the situation when multi-set (bag) semantics is considered. Now, 30 years later, decidability of QCP for bag semantics remains an open question, one of the most intriguing open questions in database theory. In this paper we show a series of undecidability results for some generalizations of this problem. We show, for example, that the problem whether, for given two boolean conjunctive queries φ s and φ b , and a linear function F, the inequality F(φ s (D)) =< φ b (D) holds for each database instance D, is undecidable.

BF-BigGraph: An efficient subgraph isomorphism approach using machine learning for big graph databases

Graph databases are flexible NoSQL databases used to efficiently store and query complex and big data. One of the most difficult problems in graph databases is the problem of subgraph isomorphism, which involves finding a matching pattern in a given graph. Subgraph isomorphism algorithms generally encounter problems in the efficient processing of complex queries based on a lack of pruning methods and the use of a matching order. In this study, we present a new subgraph isomorphism approach based on the best-first search design strategy and name it BF-BigGraph. Our approach includes a machine learning technique to efficiently find the best matching order for various complex queries. The parameters we used in our approach as heuristics to improve the performance of complex queries on graph-based NoSQL databases are database volatility, database size, type of query, and the size of the query. We utilized the Random Forest machine learning method to narrow candidate nodes to a higher level of search and effectively reduce the search space for efficient querying and retrieval. We compared BF-BigGraph with state-of-the-art approaches, namely BB-Graph, Neo4j’s Cypher, DualIso, GraphQL, TurboIso, and VF3 using publicly available databases including undirected graphs; WorldCup, Pokec, Youtube, and a big graph database of a real demographic application (a population database) with approximately 70 million nodes of a big directed graph. The performance results of our approach for different types of complex queries on all these databases are significantly better in terms of computation time and required memory than other competing approaches in the literature.

Affiliate Automation Project

The emergence of chatbots has revolutionized customer engagement and sales strategies for e-commerce platforms like Amazon. This project focuses on developing a Telegram chatbot specifically tailored for Amazon affiliate marketing. The chatbot serves as an intermediary between users and Amazon's vast product catalog, providing a seamless experience for browsing and purchasing products while integrating affiliate links to monetize interactions. Key functionalities of the chatbot include natural language processing for understanding user queries, product recommendation based on user preferences and historical data, real-time updates on product availability and pricing, and secure payment processing. The integration with Amazon's Affiliate Program ensures that affiliate links are dynamically generated and embedded within product recommendations, enabling the chatbot operator to earn commissions for referral sales.

Federated Multiple Tensor-on-Tensor Regression (FedMTOT) for Multimodal Data Under Data-Sharing Constraints

In recent years, diversified measurements reflect the system dynamics from a more comprehensive perspective in system modeling and analysis, such as scalars, waveform signals, images, and structured point clouds. To handle such multimodal structured high-dimensional (SHD) data, combining a large amount of data from multiple sites is necessary (i) to reduce the inherent population bias from a single site and (ii) to increase the model accuracy. However, impeded by data management policies and storage costs, data could not be easily shared or directly exchanged among different sites. Instead of simplifying or facilitating the data query process, we propose a federated multiple tensor-on-tensor regression (FedMTOT) framework to train the individual system model locally using (i) its own data and (ii) data features (not data itself) from other sites. Specifically, federated computation is executed based on alternating direction method of multipliers (ADMM) to satisfy data-sharing requirements, while the individual model at each site can still benefit from feature knowledge from other sites to improve its own model accuracy. Finally, two simulations and two case studies validate the superiority of the proposed FedMTOT framework.

IDify - Distributed Database System for Digital Identification

”IDify” Distributed database system for digital identification is a big data handling system to centralize the important services provided by the government to make the facilities efficient and reliable. It has integrated services such as digital authentication, E-Banking, Personal Information Management, and Government Surveys to provide users with a centralized and digitized platform to make their lives easier. In this system there is a distributed database platform to carry out all the data storing and data processing tasks to facilitate a large user count and to avoid the drawbacks of monolithic systems. The database system has been designed by integrating several modern database technologies to make the system efficient and coherent as the facilities provided by the system are extremely mission-critical. To make the system low latent on data ingestion and data reading tasks a special library is also implemented in collaboration with modern query processing engines. It is proposed to utilize the thread system of a processor to implement parallelism to enhance the performance of data read/write tasks. Apart from that a fault-tolerant framework has been implemented to free the system from any erroneous scenario taking place.

Thorough Data Pruning for Join Query in Database System

The improvement of robustness and efficiency for multi-way equijoin query is challenging, no-matter for centralized database systems or distributed database systems. Due to lots of unnecessary data existing during query processing, these two metrics will be seriously reduced. If we can thoroughly prune unnecessary data in advance, the robustness and efficiency will be highly improved. However, the pruning power of current strategies, such as predicate push-down and algebraic equivalence, is limited. We present deepDP, a powerful, generalized, and efficient strategy for data pruning. deepDP builds multiple independent pruning spaces by generating longest transitive closures and applies appropriate data pruning strategy for each pruning space. For thoroughly pruning unnecessary data, deepDP employs <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\alpha \cdot \beta$</tex-math></inline-formula> pruning strategy to clean each pruning space based on a newly designed statistic information-Hollow Range and re-shuffles the elements in all pruned spaces for maximizing robustness and efficiency benefits meanwhile minimizing the invasion. We implement deepDP in PostgreSQL but are not limited to it, and evaluate deepDP on TPC-H, JOB, and our synthesis benchmark–DHR. The experiment results show that compared to traditional data pruning strategy, deepDP can improve multi-way equijoin query on efficiency by 3.5x.

Time-Restricted, Verifiable, and Efficient Query Processing Over Encrypted Data on Cloud

Time-Restricted, Verifiable, and Efficient Query Processing Over Encrypted Data on Cloud

Design of an RFID-Based Self-Jamming Identification and Sensing Platform

Commodity RFID tags backscatter stored electronic product code (EPC) to the reader, but do not have sensing capability. Existing works have made much effort on designing RFID-based sensing platform. But most of them either need intricate hardware design or rely on modification of the tag, which increases the cost or constrains the sensing capability. In this paper, we design a self-jamming identification and sensing platform (SJISP) consisting of SJISP nodes and a commodity RFID reader. A subtle design of the SJISP node is the adoption of a jammer radio module with the same frequency as the reader, controlled by the micro control unit (MCU) to decide whether to interfere with the query process of the RFID reader. The RFID tag is not readable if the jammer is turned on to generate interference signals. Otherwise, it is readable when the jammer is turned off. The sensing data is thus modulated by switching the jammer on and off for transmitting bit 0 and bit 1, respectively. The reader demodulates the data through the compatible EPC UHF Gen2 air interface protocol. To further save the energy of the SJISP node, we propose a prefix codebook based data delivery scheme, which leverages the difference of energy consumption (DEC) between transmitting bit 0 and bit 1. Our proposed scheme can save more than 50 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\%$</tex-math></inline-formula> of the energy than common communication without codebook. Experimental results based on our prototyped system show that the designed SJISP can achieve an average packet reception rate (PRR) of over 99 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\%$</tex-math></inline-formula> and is quite robust to environmental disturbance. Our designed platform provides a low-cost and compatible solution to extend the sensing capability of RFID system. A demo application with a temperature sensor and a light sensor embedded in two SJISP nodes respectively are developed to demonstrate how SJISP applies in real world scenario.

FFO: Forest Fire Ontology and Reasoning System for Enhanced Alert and Management Services

Forest fires or wildfires pose a serious threat to property, lives, and the environment. Early detection and mitigation of such emergencies, therefore, play an important role in reducing the severity of the impact caused by wildfire. Unfortunately, there is often an improper or delayed mechanism for forest fire detection which leads to destruction and losses. These anomalies in detection can be due to defects in sensors or a lack of proper information interoperability among the sensors deployed in forests. This paper presents a lightweight ontological framework to address these challenges. Interoperability issues are caused due to heterogeneity in technologies used and heterogeneous data created by different sensors. Therefore, through the proposed Forest Fire Detection and Management Ontology (FFO), we introduce a standardized model to share and reuse knowledge and data across different sensors. The proposed ontology is validated using semantic reasoning and query processing. The reasoning and querying processes are performed on real-time data gathered from experiments conducted in a forest and stored as RDF triples based on the design of the ontology. The outcomes of queries and inferences from reasoning demonstrate that FFO is feasible for the early detection of wildfire and facilitates efficient process management subsequent to detection.

A learned cost model for big data query processing

The efficiency of query processing in the Spark SQL big data processing engine is significantly affected by execution plans and allocated resources. However, existing cost models for Spark SQL rely on hand-crafted rules. While learning-based cost models have been proposed for relational databases, they do not consider available resources. To address this issue, we propose a resource-aware deep learning model capable of automatically predicting query plan execution times based on historical data. To train our model, we embed query execution plans within a query plan tree and extracted features from allocated resources. An adaptive attention mechanism is integrated into the deep learning model to enhance prediction accuracy. Additionally, we extract sufficient features to represent data information and learn the effect of the data on query execution. This approach reduces the need for model retraining owing to data changes. The experimental results demonstrate that our deep cost model outperforms traditional rule-based methods and relational database learning-based optimizers in predicting query plan execution times.

CDFRS: A scalable sampling approach for efficient big data analysis

The sampling-based approximation method has demonstrated its potential in various domains such as machine learning, query processing, and data analysis. Most preceding sampling algorithms generate samples at the record level, making it impractical to apply them to very large datasets using a single machine. Even distributed solutions encounter efficiency issues when dealing with terabyte-scale datasets. In this paper, we introduce a scalable sampling approach named CDFRS, which can generate samples with a distribution-preserving guarantee from extensive datasets. CDFRS exhibits significantly improved speed compared to existing sampling algorithms when dealing with terabyte-scale datasets. We provide theoretical guarantees and empirical justifications, demonstrating that samples generated by the CDFRS approach maintain the distribution characteristics of the original dataset. Additionally, we propose a sample size determination algorithm, denoted as A2. Experiment results indicate that the running time of CDFRS shows at least an order of magnitude improvement over other distributed sampling methods. Notably, sampling a 10TB dataset using CDFRS only takes hundreds of seconds, while the compared method requires more than ten thousand seconds. In the context of big data analysis, including tasks such as classification and clustering, models trained with samples generated by CDFRS closely match those trained with the entire training set. Furthermore, the proposed A2 algorithm efficiently determines an appropriate sample size compared with traditional methods.

StreamFilter: a framework for distributed processing of range queries over streaming data with fine-grained access control

StreamFilter: a framework for distributed processing of range queries over streaming data with fine-grained access control

Publisher Correction: Improving query processing in blockchain systems by using a multi-level sharding mechanism

Publisher Correction: Improving query processing in blockchain systems by using a multi-level sharding mechanism

Efficient processing of coverage centrality queries on road networks

Coverage Centrality is an important metric to evaluate vertex importance in road networks. However, current solutions have to compute the coverage centrality of all the vertices together, which is resource-wasting, especially when only some vertices centrality is required. In addition, they have poor adaption to the dynamic scenario because of the computation inefficiency. In this paper, we focus on the coverage centrality query problem and propose a method that efficiently computes the centrality of single vertices without relying on the underlying graph being static by employing the intra-region pruning, inter-region pruning, and top-down search. We further propose the bottom-up search and mixed search to improve efficiency. Experiments validate the efficiency and effectiveness of our algorithms compared with the state-of-the-art method.

InferDB: In-Database Machine Learning Inference Using Indexes

The performance of inference with machine learning (ML) models and its integration with analytical query processing have become critical bottlenecks for data analysis in many organizations. An ML inference pipeline typically consists of a preprocessing workflow followed by prediction with an ML model. Current approaches for in-database inference implement preprocessing operators and ML algorithms in the database either natively, by transpiling code to SQL, or by executing user-defined functions in guest languages such as Python. In this work, we present a radically different approach that approximates an end-to-end inference pipeline (preprocessing plus prediction) using a light-weight embedding that discretizes a carefully selected subset of the input features and an index that maps data points in the embedding space to aggregated predictions of an ML model. We replace a complex preprocessing workflow and model-based inference with a simple feature transformation and an index lookup. Our framework improves inference latency by several orders of magnitude while maintaining similar prediction accuracy compared to the pipeline it approximates.

Innovative Approaches to Full-Text Search with Solr and Lucene

Full-text search engines help efficiently process large volumes of textual material and provide appropriate results. Apache Solr and Apache Lucene are popular full-text search tools for indexing and querying huge datasets. This research study examines Solr and Lucene's strengths, weaknesses, and unique methods for improving full-text search efficiency and accuracy. Apache Lucene, the fundamental full-text search framework, has extensive indexing and querying features. Developers may tailor the search process using its flexible and extendable framework. Advanced indexing algorithms like inverted indices and tokenization underpin Lucene's search capabilities. However, complicated query needs and efficient large-scale data management remain problems. Solr, founded on Lucene, adds faceting, distributed searching, and rich text analysis to its search engine. Enterprise applications may use Solr's high availability, fault tolerance, and large-scale deployments. Solr has performance tuning and setup complexity issues despite these benefits. This study explores novel solutions to these issues and improves full-text search. Advanced tokenization and normalization may improve indexing tactics. Machine learning algorithms increase search relevancy, providing more accurate and contextual results. Query processing optimization is another invention. Caching, query rewriting, and parallel processing may minimize query latency and boost throughput. GPUs are also used to improve query execution. The article also discusses integrating Solr and Lucene with big data platforms and cloud services. Distributed computing frameworks and cloud storage may improve scalability and real-time search. How Solr and Lucene may incorporate AI and NLP to improve search accuracy and user experience is also investigated.

Improving query processing in blockchain systems by using a multi-level sharding mechanism

Improving query processing in blockchain systems by using a multi-level sharding mechanism

Privacy-Preserving and Ranked Search Using Advanced Multi-Keyword Scheme Over the Encrypted Cloud Environment

In order to enable users to extract useful insights on demand, this paper presents a unique approach for secure and privacy-preserving ranked search in cloud environments. There is a rising need for strong systems that strike a balance between data utility and individual privacy as cloud-based data processing and storage become more and more prevalent. Our suggested methodology makes use of sophisticated multi-keyword operators and encryption methods to offer a safe and effective solution for cloud-based ranked search that protects privacy. Our methodology's essential component is the combination of a safe encryption technique and sophisticated multi-keyword search operators. With this combination, users can do ranked searches on encrypted data while maintaining the privacy of the underlying data. Our approach leverages strong cryptography protocols and an advanced indexing mechanism to provide rapid retrieval of relevant results while protecting confidential information from unauthorized individuals. Secure query processing techniques and encryption are added for added privacy protection. This guarantees that throughout the ranking search operations, the decrypted data cannot be accessed by the cloud service provider or any other parties participating in the data processing pipeline. Users don't have to compromise the privacy of their stored data or expose the exact text of their queries to obtain ranked results based on several keywords. Our technology is especially useful in situations when customers need instantaneous insights from data stored in the cloud without compromising privacy. Our technology protects the secrecy of the underlying datasets while enabling customers to obtain secure insights on demand in a variety of sensitive sectors, including financial analysis and medical research. The suggested method offers a strong and adaptable way to achieve privacy-preserving ranked search in cloud environments. Through the incorporation of sophisticated multi-keyword operators and encryption strategies, our approach guarantees users may effectively obtain pertinent data while maintaining personal privacy. With its ability to provide safe, anytime access to insights stored in cloud-based data repositories, this breakthrough has enormous potential for a variety of applications.

Accelerating range minimum queries with ray tracing cores

Over the past decade, GPU technology has undergone a notable transformation, evolving from pure general-purpose computation to the integration of application-specific integrated circuits (ASICs), including Tensor Cores and Ray Tracing (RT) cores. While these specialized GPU cores were initially developed to enhance specific domains like AI and real-time rendering, recent research has successfully harnessed their capabilities to expedite other tasks traditionally reliant on conventional GPU computing. One GPU task that is still yet to find its way into RT cores is the processing of range minimum queries (RMQs) in parallel, which is fundamental in fields such as information retrieval or pattern matching, among others. In this context, accelerating RMQs with RT cores would impact many of the applications that heavily rely on this task. In this work we present RTXRMQ, a new approach that can compute RMQs with RT cores. The main contribution is the proposal of a geometric solution for RMQ, where elements become triangles that are placed and shaped according to the element’s value and position in the array, respectively, such that the closest hit of a ray launched from a point given by the query parameters corresponds to the result of that query. Experimental results show that RTXRMQ is currently best suited for small query ranges relative to the input size, achieving up to 5× and 2.3× of speedup over parallel state of the art CPU and GPU approaches, respectively. For medium and large query ranges RTXRMQ is still slower than the state of the art GPU approach, but still competitive by being 2.5× and 4× faster than a state of the art CPU method running in parallel as well. Furthermore, performance scaling experiments across the latest RTX GPU architectures show that if the current RT core scaling trend continues, then RTXRMQ’s performance would scale at a higher rate than the other compared approaches, making it an attractive tool for future high performance applications that employ many batches of RMQs.

IB-CBB: an improved spatial index considering intersection based on clipped bounding boxes

ABSTRACT Efficiently querying multiple spatial datasets is a challenging task in geoscience. The majority of spatial processing techniques use minimum bounding box (MBB) to approximate neighbouring spatial objects and and place them adjacent in the spatial index. However, due to the existence of redundant space in MBB of these methods, this problem can significantly reduce the query efficiency. In this paper, we propose a novel two-stage adaptive method of clipping the bounding box in spatial query, called IB-CBB (Intersection Based Clipped Bounding Boxes). The first stage employs a clipped bounding box, which records the redundant spatial spaces within the bounding box of the spatial index by calculating the clip points. As a result, the computational complexity of indexed child nodes in the query process is reduced. The second stage optimizes the above query algorithm by judging the intersection of the query box and the MBB of index node, significantly reducing the query time. Experiments demonstrate that IB-CBB outperforms the baseline method in terms of reducing the computational time.