Query Optimization Techniques Research Articles

Rough relational database and query optimization techniques.

Rough relational database and query optimization techniques.

Analysis of Query Optimization Techniques in Databases

Abdullah Dilsat : Query Optimization in Distributed Databases. Report, Middle East Technical University, December 2003. Aho, A. V. , Sagiv,Y. and J. D. Ullman: Efficient optimization of a class of relational expressions. ACM Trans. Database Systems. 4, 4, p435-454, 1979. Chaudhuri S. and K. Shim. Query optimization with aggregate views. In Proceedings of the 5th International Conference on Extending Database Technology, Avignon, France, March 1996.

Performance Evaluation of Indices-Based Query Optimization from Flash-Based Data Centric Sensor Devices in Wireless Sensor Networks

Flash memory has become a more widespread storage medium for modern wireless devices because of its effective characteristics like nonvolatility, small size, light weight, fast access speed, shock resistance, high reliability, and low power consumption. Sensor nodes are highly resource constrained in terms of limited processing speed, runtime memory, persistent storage, communication bandwidth, and finite energy. Therefore, for wireless sensor networks supporting sense, store, merge, and send schemes, an energy efficient and reliable database-based query optimization technique is highly required with consideration of sensor node constraints. Databases on hard disk drives perform data storage and retrieval using index structures which are still not practiced for sensor devices. In this paper, we evaluate different indices like B-tree, R-tree, and MR-tree by implementing them on log structured external NAND flash memory-based advanced file systems for supporting energy efficient data storage and query optimization from flash based data centric sensor devices in wireless sensor networks. Experimental results show that PIYAS (Rizvi and Chung, 2010) file system along with B-tree indexing deployed on flash memory MLC gives the significant performance in respect of high query throughput optimization and less resources consumption for wireless sensor devices.

Обзор развития методов лексической оптимизации запросов

The presented overview is concerned with lexical query optimization and covers papers published in the last four decades. The paper consists of five sections. The first section – Introduction – classifies query optimization techniques into semantic optimizations and lexical optimizations. Semantic optimizations usually relies on data integrity rules that are stores within metadata part of databases, and on data statistics. This kind of optimizations is discussed in many textbooks and papers. Lexical optimizations (more often called rewriting) use only a text of query and no other information about database and its structure. Lexical optimizations are further classified into query transformations, query amelioration, and query reduction. The second section of the paper discusses techniques of query transformation such as predicate pushdown, transformation of nested query into query with joins, etc. Query amelioration is a topic of the third section with a focus on magic set optimizations. The forth section covers query reduction optimizations. The section briefly describes traditional approaches (such as tableau -based) are briefly described and considers in more details three new algorithms proposed by authors. The fifth section concludes the paper.

New Database Architecture for Smart Query Handler of Spatial Database

A spatial database system is a database system with additional capabilities for handling spatial data. It also supports spatial data types in its implementation, providing spatial indexing and efficient algorithms for spatial join. The retrieval of data values from a spatial database involves searching through the huge repository of data, involving huge cost. Thus query optimization on spatial database takes more time as compared to RDBMS. The current state of art shows that during the execution of a query in a spatial database management system (SDBMS), the query optimizer creates all possible query evaluation plans. All plans are equivalent in term of their final output but vary in their execution cost, the amount of time to run. Once the data is retrieved the query and its plans are deleted from memory to free the space for future usage. This is repeated for the next query even if the query is already executed. This leads to increased storage overhead and execution time. In this paper, a new database architecture is proposed, which uses a buffer based query optimization technique for faster data retrieval.

Data storage practices and query processing in XML databases: A survey

With the rapid emergence of XML as a data exchange standard over the Web, storing and querying XML data have become critical issues. The two main approaches to storing XML data are (1) to employ traditional storage such as relational database, object-oriented database and so on, and (2) to create an XML-specific native storage. The storage representation affects the efficiency of query processing. In this paper, firstly, we review the two approaches for storing XML data. Secondly, we review various query optimization techniques such as indexing, labeling and join algorithms to enhance query processing in both approaches. Next, we suggest an indexing classification scheme and discuss some of the current trends in indexing methods, which indicate a clear shift towards hybrid indexing.

Dynamic Magic Sets and super-coherent answer set programs

For many practical applications of ASP, for instance data integration or planning, query answering is important, and therefore query optimization techniques for ASP are of great interest. Magic Sets are one of these techniques, originally defined for Datalog queries (ASP without disjunction and negation). Dynamic Magic Sets (DMS) are an extension of this technique, which has been proved to be sound and complete for query answering over ASP programs with stratified negation. A distinguishing feature of DMS is that the optimization can be exploited also during the non-deterministic phase of ASP engines. In particular, after some assumptions have been made during the computation, parts of the program may become irrelevant to a query under these assumptions. This allows for dynamic pruning of the search space, which may result in exponential performance gains. In this paper, the correctness of DMS is formally established and proved for brave and cautious reasoning over the class of super-coherent ASP programs (ASP sc programs). ASPsc programs guarantee consistency (i.e., have answer sets) when an arbitrary set of facts is added to them. This result generalizes the applicability of DMS, since the class of ASPsc programs is richer than ASP programs with stratified negation, and in particular includes all odd-cycle-free programs. DMS has been implemented as an extension of DLV, and the effectiveness of DMS for ASPsc programs is empirically confirmed by experimental results with this system.

Secure XML querying based on authorization graphs

XML is rapidly emerging as a standard for data representation and exchange over the World Wide Web and an increasing amount of sensitive business data is processed in XML format. Therefore, it is critical to have control mechanisms to restrict a user to access only the parts of XML documents that she is authorized to access. In this paper, we propose the first DTD-based access control model that employs graph matching to analyze if an input query is fully acceptable, fully rejectable, or partially acceptable. In this way, there will be no further security overhead for the processing of fully acceptable and rejectable queries. For partially acceptable queries, we propose a graph-matching based authorization model for an optimized rewriting procedure in which a recursive query (query with descendant axis `//') will be rewritten into an equivalent recursive one if possible and into a non-recursive one only if necessary, resulting queries that can fully take advantage of structural join based query optimization techniques. Moreover, we propose an index structure for XML element types to speed up the query rewriting procedure, a facility that is potentially useful for applications with large DTDs. Our performance study results showed that our algorithms armed with rewriting indexes are promising.

SmartCIS

With the increasing adoption of networked sensors, a new class of applications is emerging that combines data from the "digital world" with real-time sensor readings, in order to intelligently manage physical environments and systems (e.g., "smart" buildings, power grids, data centers). This leads to new challenges in providing programmability, performance, extensibility, and multi-purpose heterogeneous data acquisition. The ASPEN project addresses these challenges by extending data integrationtechniques to the distributed stream world, and adding new abstractions for physical phenomena. We describe the architecture and implementation of our ASPEN system and its showcase intelligent building application, SmartCIS, which was demonstrated at SIGMOD 2009. We summarize the new query processing algorithms we have developed for integrating highly distributed stream data sources, both in low-power sensor devices and traditional PCs and servers; describe query optimization techniques for federations of stream processors; and detail new capabilities such as incremental maintenance of recursive views. Our algorithms and techniques generalize across a wide range of data from RFID and light measurements to real-time machine usage monitoring, energy consumption and recursive query computation.

Building disclosure risk aware query optimizers for relational databases

Many DBMS products in the market provide built in encryption support to deal with the security concerns of the organizations. This solution is quite effective in preventing data leakage from compromised/stolen storage devices. However, recent studies show that a significant part of the leaked records have been done so by using specialized malwares that can access the main memory of systems. These malwares can easily capture the sensitive information that are decrypted in the memory including the cryptographic keys used to decrypt them. This can further compromise the security of data residing on disk that are encrypted with the same keys. In this paper we quantify the disclosure risk of encrypted data in a relational DBMS for main memory-based attacks and propose modifications to the standard query processing mechanism to minimize such risks. Specifically, we propose query optimization techniques and disclosure models to design a data-sensitivity aware query optimizer. We implemented a prototype DBMS by modifying both the storage engine and optimizer of MySQL-InnoDB server. The experimental results show that the disclosure risk of such attacks can be reduced dramatically while incurring a small performance overhead in most cases.

Real-Time Query Processing on Live Videos in Networks of Distributed Cameras

With the proliferation of inexpensive cameras and the availability of high-speed wired and wireless networks, systems of distributed cameras are becoming an enabling technology for a broad range of interdisciplinary applications in domains such as public safety and security, manufacturing, transportation, and healthcare. Today’s live video processing systems on networks of distributed cameras, however, are designed for specific classes of applications. To provide a generic query processing platform for applications of distributed camera networks, the authors designed and implemented a new class of general purpose database management systems, the live video database management system (LVDBMS). The authors view networked video cameras as a special class of interconnected storage devices, and allow the user to formulate ad hoc queries expressed over real-time live video feeds. This paper introduces their system and presents the live video data model, the query language, and the query processing and optimization technique.

PDCS: Security and Privacy Support for Data-Centric Sensor Networks

The demand for efficient data dissemination/access techniques to find relevant data from within a sensor network has led to the development of data-centric sensor (DCS) networks, where the sensor data instead of sensor nodes are named based on attributes such as event type or geographic location. However, saving data inside a network also creates security problems due to the lack of tamper resistance of the sensor nodes and the unattended nature of the sensor network. For example, an attacker may simply locate and compromise the node storing the event of his interest. To address these security problems, we present pDCS, a privacy-enhanced DCS network which offers different levels of data privacy based on different cryptographic keys. pDCS also includes an efficient key management scheme to facilitate the management of multiple types of keys used in the system. In addition, we propose several query optimization techniques based on Euclidean Steiner tree and keyed bloom filter (KBF) to minimize the query overhead while preserving query privacy. Finally, detailed analysis and simulations show that the KBF scheme can significantly reduce the message overhead with the same level of query delay and maintain a very high level of query privacy.

An intelligent query processing for distributed ontologies

In this paper, we propose an intelligent distributed query processing method considering the characteristics of a distributed ontology environment. We suggest more general models of the distributed ontology query and the semantic mapping among distributed ontologies compared with the previous works. Our approach rewrites a distributed ontology query into multiple distributed ontology queries using the semantic mapping, and we can obtain the integrated answer through the execution of these queries. Furthermore, we propose a distributed ontology query processing algorithm with several query optimization techniques: pruning rules to remove unnecessary queries, a cost model considering site load balancing and caching, and a heuristic strategy for scheduling plans to be executed at a local site. Finally, experimental results show that our optimization techniques are effective to reduce the response time.

Query translation from XPath to SQL in the presence of recursive DTDs

We study the problem of evaluating xpath queries over xml data that is stored in an rdbms via schema-based shredding. The interaction between recursion (descendants-axis) in xpath queries and recursion in dtds makes it challenging to answer xpath queries using rdbms. We present a new approach to translating xpath queries into sql queries based on a notion of extended XP ath expressions and a simple least fixpoint (lfp) operator. Extended xpath expressions are a mild extension of xpath, and the lfp operator takes a single input relation and is already supported by most commercial rdbms. We show that extended xpath expressions are capable of capturing both dtd recursion and xpath queries in a uniform framework. Furthermore, they can be translated into an equivalent sequence of sql queries with the lfp operator. We present algorithms for rewriting xpath queries over a (possibly recursive) dtd into extended xpath expressions and for translating extended xpath expressions to sql queries, as well as optimization techniques. The novelty of our approach consists in its capability to answer a large class of xpath queries by means of only low-end rdbms features already available in most rdbms, as well as its flexibility to accommodate existing relational query optimization techniques. In addition, these translation algorithms provide a solution to query answering for certain (possibly recursive) xml views of xml data. Our experimental results verify the effectiveness of our techniques.

A Study On Query Optimization for Federated Database Systems

<!-- /* Font Definitions */ @font-face {font-family:??; panose-1:2 1 6 0 3 1 1 1 1 1; mso-font-alt:SimSun; mso-font-charset:134; mso-generic-font-family:auto; mso-font-pitch:variable; mso-font-signature:3 135135232 16 0 262145 0;} @font-face {font-family:

A Multi-Layered Approach to the Design of Intelligent Intrusion Detection and Prevention System (IIDPS)

Introduction Considering the reliance of humans on computers and network infrastructures to perform virtually every aspect of day to day activity, there is a critical need for ensuring the reliability and integrity of these infrastructures. According to the National Institute of Standards and Technology, Intrusion is an attempt to compromise the confidentiality, integrity, availability or an attempt to bypass the security mechanisms of a computer or network (Jones & Sielkens, 2000). The reasons for these intrusions could be attempts to steal a company's most valuable information, personal employee and customer information or to use the company's computer resources, etc. For example, the 2003 CSI/FBI (Computer Security Institute/ Federal Bureau of Investigation) Computer Crime and Security Survey reported that participants in the survey lost about $135 million from the theft of proprietary information and denial of service attacks (Cisco Systems, 2004). Recently, Intrusion Detection Systems (IDS) have been used in monitoring attempts to break security, which provides important information for timely countermeasures (Chen, Abraham, & Yang, 2007). Intrusion Detection System (IDS) implements application monitors in the form of a software program to learn and monitor the behavior of system programs in order to detect attacks against computer hosts. Existing IDSs are built with either signature-based or anomaly-based system, Signature matching is based on a misuse model, this intrusion detection system detects intrusions by looking for activities that corresponds to known intrusion techniques(signatures)or system vulnerabilities while anomaly detection is based on a normal use model (Hwang, Cai, Chen, & Qin, 2007), they detect intrusion by looking for activities that is different from a user's or systems normal behavior. They may be classified into Host-based and Network-based according to the information used by each IDS. A Host-based IDS refers to the class of intrusion that resides on the monitor and the individual host machine, while A Network-based IDS monitors the packets that traverse a given network link (Jones & Sielken, 2000). The system proposed here is a type Host-based intrusion detection systems (HIDSs),these type of systems rely on events collected by the host they monitor .HIDSs can be classified based on the type of audit data they analyze or based on the techniques used to analyze their input. Common classes: * operating system-level intrusion detection systems * application-level intrusion detection systems The system proposed here is an operating system-level intrusion system, because the OS is a trusted entity and it controls access to resources, such as memory and files. Overview of Existing Systems From the literature, there are various works in the field of intrusion detection system. This paper reviews those that are closely related to the proposed work based on the Anomaly and Signature detection approaches and the combination of both. In order to ascertain the efficiency of the new approach, a comparison is drawn between the existing work that have used the combination of both approaches and the new system which shows further improvement over the existing ones. Adaptable Real-time Misuse Detection System (ARMD) (1998) is a host-based misuse detection system. Its pattern of signatures is over a sequence of abstract events and this is tagged MuSig'. This describes conditions that the abstract event attributes must satisfy. Based on the signatures (MuSigs), the available audit trail, and the strategy costs, ARMD uses a strategy generator to automatically generate monitoring strategies to govern the misuse detection process. It employs database query optimization techniques to speed up the processing of audit events. One advantage of ARMD is that knowing the characteristics of the audit trail helps estimate the cost of performing misuse detection and this gives the security officers the opportunity to tune the misuse detection system. …

Using Wide Table to manage web data: a survey

With the development of World Wide Web (www), storage and utilization of web data has become a big challenge for data management research community. Web data are essentially heterogeneous data, and may change schema frequently, traditional relational data model is inappropriate for web data management. A new data model, called Wide Table (or WT for simplicity), was introduced for this task. There are several characteristics of the WT model. First, WT is usually highly sparsely populated so that most data can be fit into a line or record. Second, queries are composed on only a small subset of the attributes. Thus, existing query processing and optimization techniques for relational database with normalized tables will not work efficiently anymore. Furthermore, WT is usually of extremely large volume. It is thought that only large-scale distributed storage can accommodate themassive data set. In this paper, requirements and challenges to web data management are discussed. Existing techniques for WT, including logical presentation, physical storage, and query processing, are introduced and analyzed in detail.

Performance profiling with EndoScope, an acquisitional software monitoring framework

We propose EndoScope, a software monitoring framework that allows users to pose declarative queries that monitor the state and performance of running programs. Unlike most existing monitoring tools, EndoScope is acquisitional , meaning that it only instruments the portions of the program that need to be monitored to answer queries. The use of a high level declarative language allows EndoScope to search for efficient physical instantiations of queries by applying a suite of optimizations, including control flow graph analysis, and traditional database query optimization techniques, such as predicate pushdown and join optimization, to minimize the number of program instrumentation points and overhead to the monitored program. Furthermore, a flexible, high level language and the ability to attach to running programs enable developers to build various program analysis and monitoring applications beyond traditional software profilers with EndoScope. We describe a prototype implementation of the EndoScope framework and a simple profiler for Java programs implemented with EndoScope. We show results from using our profiler on a collection of real-world programs, including a TPC-C implementation using the Derby database and the petstore application running on top of Tomcat application server. Our results show the benefit of our optimization framework and demonstrate that our declarative, acquisitional approach can yield program instrumentation overheads that are dramatically lower than conventional profiling tools (for example, when profiling the Derby Database running TPC-C, our system's overhead ranges from 1% to about 25%, whereas the fastest existing profiler we measured imposes a minimum overhead of about 30%.)

Query Processing and Optimization Techniques over Streamed Fragmented XML

With the extensive use of XML in applications over the Web, efficient query processing over streaming XML has become a core challenge due to one-pass processing and limited resources. Taking advantage of Hole-Filler model for XML fragments, this paper proposes a hybrid structure (FQ-Index) for both the queries and fragments, and proposes an XML fragment processing algorithm to evaluate forward XPath queries over streamed XML fragments. Two optimization rules, dependence pruning and prefix pruning are also developed. Dependence pruning scheme prunes off the dependent operations caused by fragmentation and transforms the queries for XML tag into queries for XML fragments, while prefix pruning scheme prunes off the "redundant" prefix along the path according to the tag structure. The effectiveness of the techniques developed is illustrated with a detailed set of experiments.

Query evaluation and optimization in the semantic web

AbstractWe address the problem of answering Web ontology queries efficiently. An ontology is formalized as adeductive ontology base(DOB), a deductive database that comprises the ontology's inference axioms and facts. A cost-based query optimization technique for DOB is presented. A hybrid cost model is proposed to estimate the cost and cardinality of basic and inferred facts. Cardinality and cost of inferred facts are estimated using an adaptive sampling technique, while techniques of traditional relational cost models are used for estimating the cost of basic facts and conjunctive ontology queries. Finally, we implement a dynamic-programming optimization algorithm to identify query evaluation plans that minimize the number of intermediate inferred facts. We modeled a subset of the Web ontology language Lite as a DOB and performed an experimental study to analyze the predictive capacity of our cost model and the benefits of the query optimization technique. Our study has been conducted over synthetic and real-world Web ontology language ontologies and shows that the techniques are accurate and improve query performance.