Query Processing Optimization Research Articles

Innovative Approaches to Full-Text Search with Solr and Lucene

Full-text search engines help efficiently process large volumes of textual material and provide appropriate results. Apache Solr and Apache Lucene are popular full-text search tools for indexing and querying huge datasets. This research study examines Solr and Lucene's strengths, weaknesses, and unique methods for improving full-text search efficiency and accuracy. Apache Lucene, the fundamental full-text search framework, has extensive indexing and querying features. Developers may tailor the search process using its flexible and extendable framework. Advanced indexing algorithms like inverted indices and tokenization underpin Lucene's search capabilities. However, complicated query needs and efficient large-scale data management remain problems. Solr, founded on Lucene, adds faceting, distributed searching, and rich text analysis to its search engine. Enterprise applications may use Solr's high availability, fault tolerance, and large-scale deployments. Solr has performance tuning and setup complexity issues despite these benefits. This study explores novel solutions to these issues and improves full-text search. Advanced tokenization and normalization may improve indexing tactics. Machine learning algorithms increase search relevancy, providing more accurate and contextual results. Query processing optimization is another invention. Caching, query rewriting, and parallel processing may minimize query latency and boost throughput. GPUs are also used to improve query execution. The article also discusses integrating Solr and Lucene with big data platforms and cloud services. Distributed computing frameworks and cloud storage may improve scalability and real-time search. How Solr and Lucene may incorporate AI and NLP to improve search accuracy and user experience is also investigated.

View Materialization for Query Processing in IoT Systems

In a highly dynamic environment like the Internet of Things, data is being continuously generated by IoT devices. These amounts of data make its processing and querying challenging tasks. Materialized views are ideal for query processing optimization by caching the results of queries. However, one of the main issues of the views materialization is their inconsistency with data sources. For this reason, we present a novel approach of query answering and data management in IoT systems. Our approach is based on new materialization and maintenance of views strategies. This last was carried out by factoring maintenance treatments of identical views and those sharing sources using a clustering technique. The results of experiments show that our approach is suitable for query processing under IoT environments.

Development of a Query processing optimization model in electronic health records systems

Development of a Query processing optimization model in electronic health records systems

Query processing optimization in broadcasting XML data in mobile communications

Todays, XML as a de facto standard is used to broadcast data over mobile wireless networks. In these networks, mobile clients send their XML queries over a wireless broadcast channel and recieve their desired XML data from the channel. However, downloading the whole XML data by a mobile device is a challenge since the mobile devices used by clients are small battery powered devices with limited resources. To meet this challenge, the XML data should be indexed in such a way that the desired XML data can be found easily and only such data can be downloaded instead of the whole XML data by the mobile clients. Several indexing methods are proposed to selectively access the XML data over an XML stream. However, the existing indexing methods cause an increase in the size of XML stream by including some extra information over the XML stream. In this paper, a new XML stream structure is proposed to disseminate the XML data over a broadcast channel by grouping and summarizing the structural information of XML nodes. By summarizing such information, the size of XML stream can be reduced and therefore, the latency of retrieving the desired XML data over a wirless broadcast channel can be reduced. The proposed XML stream structure also contains indexes in order to skip from the irrelevant parts over the XML stream. It therefore can reduce the energy consumption of mobile devices in downloading the results of XML queries. In addition, our proposed XML stream structure can process different types of XML queries and experimental results showed that it improves the performace of XML query processing over the XML data stream compared to the existing research works in terms of access and tuning times.

Query-Driven Learning for Predictive Analytics of Data Subspace Cardinality

Fundamental to many predictive analytics tasks is the ability to estimate the cardinality (number of data items) of multi-dimensional data subspaces, defined by query selections over datasets. This is crucial for data analysts dealing with, e.g., interactive data subspace explorations, data subspace visualizations, and in query processing optimization. However, in many modern data systems, predictive analytics may be (i) too costly money-wise, e.g., in clouds, (ii) unreliable, e.g., in modern Big Data query engines, where accurate statistics are difficult to obtain/maintain, or (iii) infeasible, e.g., for privacy issues. We contribute a novel, query-driven, function estimation model of analyst-defined data subspace cardinality. The proposed estimation model is highly accurate in terms of prediction and accommodating the well-known selection queries: multi-dimensional range and distance-nearest neighbors (radius) queries. Our function estimation model: (i) quantizes the vectorial query space, by learning the analysts’ access patterns over a data space, (ii) associates query vectors with their corresponding cardinalities of the analyst-defined data subspaces, (iii) abstracts and employs query vectorial similarity to predict the cardinality of an unseen/unexplored data subspace, and (iv) identifies and adapts to possible changes of the query subspaces based on the theory of optimal stopping. The proposed model is decentralized, facilitating the scaling-out of such predictive analytics queries. The research significance of the model lies in that (i) it is an attractive solution when data-driven statistical techniques are undesirable or infeasible, (ii) it offers a scale-out, decentralized training solution, (iii) it is applicable to different selection query types, and (iv) it offers a performance that is superior to that of data-driven approaches.

Bitwise dimensional co-clustering for analytical workloads

Analytical workloads in data warehouses often include heavy joins where queries involve multiple fact tables in addition to the typical star-patterns, dimensional grouping and selections. In this paper we propose a new processing and storage framework called bitwise dimensional co-clustering (BDCC) that avoids replication and thus keeps updates fast, yet is able to accelerate all these foreign key joins, efficiently support grouping and pushes down most dimensional selections. The core idea of BDCC is to cluster each table on a mix of dimensions, each possibly derived from attributes imported over an incoming foreign key and this way creating foreign key connected tables with partially shared clusterings. These are later used to accelerate any join between two tables that have some dimension in common and additionally permit to push down and propagate selections (reduce I/O) and accelerate aggregation and ordering operations. Besides the general framework, we describe an algorithm to derive such a physical co-clustering database automatically and describe query processing and query optimization techniques that can easily be fitted into existing relational engines. We present an experimental evaluation on the TPC-H benchmark in the Vectorwise system, showing that co-clustering can significantly enhance its already high performance and at the same time significantly reduce the memory consumption of the system.

Efficient ELM-Based Two Stages Query Processing Optimization for Big Data

MapReduce and its variants have emerged as viable competitors for big data analysis with a commodity cluster of machines. As an extension of MapReduce, ComMapReduce realizes the lightweight communication mechanisms to enhance the performance of query processing applications for big data. However, different communication strategies of ComMapReduce can substantially affect the executions of query processing applications. Although there is already the research work that can identify the communication strategies of ComMapReduce according to the characteristics of the query processing applications, some drawbacks still exist, such as relative simple model, too much user participation, and relative simple query processing execution. Therefore, an efficient ELM-based two stages query processing optimization model is proposed in this paper, named ELM to ELM (E2E) model. Then, we develop an efficient sample training strategy to train ourE2Emodel. Furthermore, two query processing executions based on theE2Emodel, respectively, Just-in-Time execution and Queue execution, are presented. Finally, extensive experiments are conducted to verify the effectiveness and efficiency of theE2Emodel.

Real-time query processing optimization for cloud-based wireless body area networks

Wireless body area networks (WBANs) have received a lot of attention from both academia and industry due to the increasing need of ubiquitous computing for eHealth applications, the continuous advances in miniaturization of electronic devices, and the ultra-low-power wireless technologies. In these networks, various sensors are attached either on clothes, on human body or even implanted under the skin for real-time health monitoring of patients in order to improve their independent daily lives. The energy constraints of sensors, the vital and large amount of data collected by WBAN nodes require powerful and secure storage, and a query processing mechanism that takes into account both real-time and energy constraints. This paper addresses these challenges and proposes a new architecture that combines a cloud-based WBANs with statistical modeling techniques in order to provide a secure storage infrastructure and optimize the real-time user query processing in terms of energy minimization and query latency. Such statistical model provides good approximate answers to queries with a given probabilistic confidence. Furthermore, the combination of the model with the cloud-based WBAN allows performing a query processing algorithm that uses the error tolerance and the probabilistic confidence interval as query execution criterions. The performance analysis and the experiments based on both real and synthetic data sets demonstrate that the new architecture and its underlying proposed algorithm optimize the real-time query processing to achieve minimal energy consumption and query latency, and provide secure and powerful storage infrastructure.

An efficient query processing optimization based on ELM in the cloud

Nowadays, MapReduce has emerged as a facto programming model for parallel processing of large-scale datasets with a commodity cluster of machines. MapReduce and its variants have been widely researched in the industry and academic communities. ComMapReduce further extends MapReduce by adding lightweight communication mechanisms and also enhances the efficiency of query processing applications. However, we find that the performance of query processing applications changes a lot in different communication strategies of ComMapReduce framework. It is necessary to identify the most optimal communication strategies of the query processing applications. Extreme learning machine (ELM) can exactly provide classification performance with an extremely fast training speed. Therefore, in this paper, first, we propose an efficient query processing optimization approach based on ELM in ComMapReduce framework, named ELM_CMR. Then, we design two implementations of our ELM_CMR approach to further optimize the performance of query processing applications. Finally, extensive experiments are conducted to verify the effectiveness and efficiency of our proposed ELM_CMR.

Survey of RDF Query Processing Techniques

With the advance of the semantic Web and information extraction techniques,more and more RDF data emerge on the Web.Managing massive RDF data has been a hot research topic for both academic and industrial communities.In the paper,we give a thorough analysis and comparison of the existing techniques from the view of the accumulation and organization of RDF data.Topics on query representation,query processing and query optimization are extensively discussed.We finally identify challenges and future work of the area.

Distributed time-aware provenance

The ability to reason about changes in a distributed system's state enables network administrators to better diagnose protocol misconfigurations, detect intrusions, and pinpoint performance bottlenecks. We propose a novel provenance model called Distributed Time-aware Provenance (DTaP) that aids forensics and debugging in distributed systems by explicitly representing time, distributed state, and state changes. Using a distributed Datalog abstraction for modeling distributed protocols, we prove that the DTaP model provides a sound and complete representation that correctly captures dependencies among events in a distributed system. We additionally introduce DistTape, an implementation of the DTaP model that uses novel distributed storage structures, query processing, and cost-based optimization techniques to efficiently query time-aware provenance in a distributed setting. Using two example systems (declarative network routing and Hadoop MapReduce), we demonstrate that DistTape can efficiently maintain and query time-aware provenance at low communication and computation cost.

A new approach to building histogram for selectivity estimation in query processing optimization

Recently, histograms have been considered as an effective way to produce quick approximate answers to decision support queries. They are also taken as a basic tool for data visualization and analysis. In this paper, we propose a new approach to constructing histograms for selectivity estimation in query processing optimization. Our approach uses a new criterion, i.e., aggregate error minimization, to direct the construction of the target histogram. We develop the algorithm of aggregate error minimization based histogram construction, and demonstrate the effectiveness and efficiency of the proposed approach by experiments over both real-world and synthetic datasets.

An integer programming formulation embedded in an algorithm for query processing optimization in distributed relational database systems

In this paper, an algorithm is presented which determines the combination of the project, select and equi-join relational operators and their respective processing sites, in a distributed relational database system, in response to a query or class of queries. Part of this algorithm, is an integer programming model which determines the files in the network to be joined and their respective processing sites so as to minimize the sum of the network transmission and local processing costs. The algorithm is executed and illustrated using an example.