Magic-sets Transformation Research Articles

Extending Query-Subquery Nets for Deductive Databases under the Well-Founded Semantics

ABSTRACTWe propose a method, called QSQN-WF, for evaluating queries to Datalog¬ databases under the well-founded semantics. It is the first one that is set-at-a-time and strictly goal-directed w.r.t. SLS-resolution defined by Przymusinski. These properties are important for reducing accesses to the secondary storage and redundant computations. The first property distinguishes our method from the one based on SLG-resolution by Chen, Swift, and Warren (1995) (which is tuple-at-a-time). The second property distinguishes our method from the ones based on the magic-sets transformation by Kemp, Srivastava, and Stuckey (1995) and Morishita (1996), which use magic atoms not in the most appropriate way and are not strictly goal-directed w.r.t. SLS-resolution. Our method follows SLS-resolution, with Van Gelder’s alternating fixpoint semantics on the background, but uses a query-subquery net to implement tabulation and the set-at-a-time technique, reduce redundant computations, and allow any control strategy within each iteration of the main loop. It is sound and complete w.r.t. the well-founded semantics and has PTIME data complexity.

Query optimization in deductive databases by magic-sets transformation.

Query optimization in deductive databases by magic-sets transformation.

Magic Sets and their application to data integration

Recently, effective methods model query-answering in data integration systems and inconsistent databases in terms of cautious reasoning over Datalog ¬ programs under the stable model semantics. Since this task is computationally expensive (co-NP-complete), there is a clear need of suitable techniques for query optimization, in order to make such methods feasible for data-intensive applications. We propose a generalization of the well-known Magic Sets technique to Datalog ¬ programs with (possibly unstratified) negation under the stable model semantics. Our technique produces a new program whose evaluation is more efficient (due to a smaller instantiation) in general, while preserving full query-equivalence for both brave and cautious reasoning, provided that the original program is consistent. Soundness under cautious reasoning is always guaranteed, even if the original program is inconsistent. In order to formally prove the correctness of our Magic Sets transformation, we introduce a novel notion of modularity for Datalog ¬ under the stable model semantics, which is more suitable for query answering than previous module definitions. We prove that a query on such a module can be evaluated independently from the rest of the program, while preserving soundness under cautious reasoning. Importantly, for consistent programs, both soundness and completeness are guaranteed for brave reasoning and cautious reasoning. Our Magic Sets optimization constitutes an effective method for enhancing the performance of data integration systems in which query-answering is carried out by means of cautious reasoning over Datalog ¬ programs. In fact, results of experiments in the EU project INFOMIX, show that Magic Sets are fundamental for the scalability of the system.

Efficient goal directed bottom-up evaluation of logic programs

This paper introduces a new strategy for the efficient goal directed bottom-up evaluation of logic programs. Instead of combining a standard bottom-up evaluation strategy with a Magicset transformation, the evaluation strategy is specialized for the application to Magic-set programs which are characterized by clause bodies with a high degree of overlapping. The approach is similar to other techniques which avoid re-computation by maintaining and reusing partial solutions to clause bodies. However, the overhead is considerably reduced as these are maintained implicitly by the underlying Prolog implementation. The technique is presented as a simple meta-interpreter for goal directed bottom-up evaluation. No Magic-set transformation is involved as the dependencies between calls and answers are expressed directly within the interpreter. The proposed technique has been implemented and shown to provide substantial speed-ups in applications of semantic based program analysis based on bottom-up evaluation.

Exploiting goal independence in the analysis of logic programs

This paper illustrates the use of a top-down framework to obtain goal independent analyses of logic programs, a task which is usually associated with the bottom-up approach. While it is well known that the bottom-up approach can be used, through the magic set transformation, for goal dependent analysis, it is less known that the top-down approach can be used for goal independent analysis. The paper describes two ways of doing the latter. We show how the results of a goal independent analysis can be used to speed up subsequent goal dependent analyses. However this speed-up may result in a loss of precision. The influence of domain characteristics on this precision is discussed and an experimental evaluation using a generic top-down analyzer is described. Our results provide intuition regarding the cases where a two phase analysis might be worth-while.

An Extension of Van Gelder's Alternating Fixpoint to Magic Programs

We study the application of the magic-sets transformation technique to Datalog⌝(function-free programs with negation) that may not have two-valued well-founded models. In this general setting, the well-founded model of the original program does not always agree with the well-founded model of the magic program derived by commonly used left-to-right sideways information-passing strategies on the query. In order to correct this disagreement we present a novel method that is obtained by slightly and naturally tailoring Van Gelder's alternating fixpoint technique to any magic program.

Magic conditions

Much recent work has focused on the bottom-up evaluation of Datalog programs [Bancilhon and Ramakrishnan 1988]. One approach, called magic-sets, is based on rewriting a logic program so that bottom-up fixpoint evaluation of the program avoids generation of irrelevant facts [Bancilhon et al. 1986; Beeri and Ramakrishnan 1987; Ramakrishnan 1991]. It was widely believed for some time that the principal application of the magic-sets technique is to restrict computation in recursive queries using equijoin predicates. We extend the magic-sets transformation to use predicates other than equality ( X >10, for example) in restricting computation. The resulting ground magic-sets transformation is an important step in developing an extended magic-sets transformation that has practical utility in “real” relational databases, not only for recursive queries, but for nonrecursive queries as well [Mumick et al. 1990b; Mumick 1991].

A Framework for Testing Safety and Effective Computability

This paper presents a methodology for testing a logic program containing function symbols and built-in predicates forsafetyandeffective computability. Safety is the property that the set of answers for a given query is finite. A related issue is whether the evaluation strategy can effectively compute all answers and terminate. We consider these problems under the assumption that queries are evaluated using a fair bottom-up fixpoint computation. We also model the use of function symbols, to construct complex terms such as lists, and arithmetic operators, by considerating Datalog programs with infinite base relations over whichfiniteness constraintsandmonotonicity constraintsare imposed. One of the main results of this paper is a recursive algorithm,check–clique, to test the safety and effective computability of predicates in arbitrarily complex cliques in the predicate connection graph. This algorithm takes certain procedures as parameters, and its applicability can be strengthened by making these procedures more sophisticated. We specify the properties required of these procedures precisely, and present a formal proof of correctness for the algorithmcheck–clique. This work can be seen as providing a framework for testing safety and effective computability of recursive programs, in some ways analogous to thecapture rulesframework of Ullman. A second important contribution is a framework for analyzing programs that are produced by theMagic Setstransformation utilizingcheck–cliqueto analyze recursive cliques. The transformed program unfortunately often has a clique structure that combines several cliques of the original program. Given the complexity of algorithmcheck–clique, it is important to keep cliques as small as possible. We deal with this problem by considering cliques in an intermediate program, called theadorned program, produced by the Magic Sets transformation. The clique structure of the adorned program is similar to that of the original program, and by showing how to analyze the transformed program in terms of the cliques in the adorned program, we avoid the potentially expensive analysis of the cliques in the transformed program.

Analyzing logic programs using “ prop”-ositional logic programs and a magic wand

This paper illustrates the role of a class of “prop”-ositional logic programs in the analysis of complex properties of logic programs. Analyses are performed by abstracting Prolog programs to corresponding “prop”-ositional logic programs which approximate the original programs and have finite meanings. We focus on a groundness analysis which is equivalent to that obtained by abstract interpretation using the domain Prop. The main contribution is in the ease in which a highly efficient implementation of the analysis is obtained. The implementation is bottom-up and provides approximations of a program's success patterns. Goal-dependent information such as call patterns is obtained using a magic-set transformation. A novel compositional approach is applied so that call patterns for arbitrary goals are derived in a precise and efficient way.

Bottom-up evaluation and query optimization of well-founded models

We present a bottom-up operational procedure for computing well-founded models of allowed programs with negation. This procedure provides a practical method of handling programs that involve unstratified negation in a manner that may be mixed with other evaluation approaches, such as semi-naive evaluation and various program transformations. We define classes of programs and sideways information of passing strategies (sips) for which the magic sets transformation preserves well-founded models with respect to the query. The classes of programs and sips we consider strictly subsume those already considered in the literature, and include stratified programs (with any choice of sips), left-to-right modularly stratified programs (with left-to-right sips) and arbitrary programs (with well-founded sips). For these programs and sips, our procedure for computing well-founded models is applicable to the rewritten programs, thus allowing increased efficiency of specializing a program for a query. Finally, we describe an extension of our bottom-up operational procedure that operates on the magic sets transformation of arbitrary programs with arbitrary sips, and computes query answers with respect to the well-founded models of the original program.

Argument reduction by factoring

We identify a useful property of a program with respect to a predicate, calledfactoring. While we prove that detecting factorability is undecidable in general, we show that for a large class of programs, the program obtained by applying the Magic Sets transformation is factorable with respect to the recursive predicate. When the factoring property holds, a simple optimization of the program generated by the Magic Sets transformation results in a new program that is never less efficient than the Magic Sets program and is often dramatically more efficient, due to the reduction of the arity of the recursive predicate. We show that the concept of factoring generalizes some previously identified special cases of recursions, including separable recursions and right- and left-linear recursions.

Implementation of magic-sets in a relational database system

We describe the implementation of the magic-sets transformation in the Starburst extensible relational database system. To our knowledge this is the first implementation of the magic-sets transformation in a relational database system. The Starburst implementation has many novel features that make our implementation especially interesting to database practitioners (in addition to database researchers). (1) We use a cost-based heuristic for determining join orders (sips) before applying magic. (2) We push all equality and non-equality predicates using magic, replacing traditional predicate pushdown optimizations. (3) We apply magic to full SQL with duplicates, aggregation, null values, and subqueries. (4) We integrate magic with other relational optimization techniques. (5) The implementation is extensible . Our implementation demonstrates the feasibility of the magic-sets transformation for commercial relational systems, and provides a mechanism to implement magic as an integral part of a new database system, or as an add-on to an existing database system.

The generalized supplementary magic-sets transformation for stratified Datalog

In this paper we extend the magic-sets methods to be even applicable to stratified Datalog. The presented GSMS neg transformation method is a very natural extension of the GSMS algorithm in [12].

Efficient Evaluation of Generalized Left-linear Recursive Queries With Multivalue Binding Sets

The left-linear transformation presented by Ullman, Naughton el al. is a rule-rewriting algorithm, which is similar to the magic sets transformation. Since left-linear recursions are among the most common recursions in practice and the transformed rules can be evaluated bottom-up efficiently, the left-linear transformation has been chosen as one of the best strategies for recursive query processing by the NALL! system at Stanford and the LDL system at MCC. However, there are two strick restraints for the left-linear transformation

Magic is relevant

We define the magic-sets transformation for traditional relational systems (with duplicates, aggregation and grouping), as well as for relational systems extended with recursion. We compare the magic-sets rewriting to traditional optimization techniques for nonrecursive queries, and use performance experiments to argue that the magic-sets transformation is often a better optimization technique.

Efficient evaluation of right-, left-, and multi-linear rules

We present an algorithm for the efficient evaluation of a useful subset of recursive queries. Like the magic sets transformation, the algorithm consists of a rewriting phase followed by semi-naive bottom-up evaluation of the resulting rules. We prove that on a wide range of recursions, this algorithm achieves a factor of Ο ( n ) speedup over magic sets. Intuitively, the transformations in this algorithm achieve their performance by reducing the arity of the recursive predicates in the transformed rules.