Magic Sets Research Articles

Magic sets and stratified databases

This article considers the efficient bottom‐up query evaluation for stratified databases. We investigate the applicability of magic-set method to stratified databases containing negative body literals and show that culprit cycles cause unstratification. Based on the analysis, we present a labeling algorithm to distinguish the context for constructing magic sets, which is simpler and more efficient than the algorithms proposed by Balbin et al. [J. Logic Programming, 295‐344 (1991)]. © 1997 John Wiley & Sons, Inc.

Magic sets with full sharing

In this paper, we study the relationship between tabulation and goal-oriented bottom-up evaluation of logic programs. Differences emerge when one tries to identify features of one evaluation method in the other. We show that to obtain the same effect as tabulation in top-down evaluation, one has to perform a careful adornment in programs to be evaluated bottom-up. Furthermore, we propose an efficient algorithm to perform forward subsumption checking over adorned magic facts.

An Efficient Evaluation Technique for Non-Stratified Programs by Transformation to Explicitly Locally Stratified Programs

We give a simple transformation from normal programs with no stratification (local, weak, modular, etc.) into a subclass of the locally stratified programs, called Explicitly Locally Stratified (ELS) programs, for which there are efficient evaluation techniques. One set of predicates are generated for the true tuples and a different set of predicate are generated for the true and undefined tuples. A similar transformation is given that incorporates a magic sets like transformation. Previous approaches to magic sets transformations of unstratified programs either restricted the class of sips used or generated a program that required special treatment of the magic sets predicates. Our transformation does not suffer from these flaws.

State-saving transformations for efficient bottom-up

Redundant computations in bottomdup evaluation of logic programs and ruledbased systems can be avoided by caching intermediate joins – i.e., partial rule instantiations – for later use. Joins can be cached either at representation level by program transformation techniques introducing supplementary predicates or at implementation level by specialized implementation techniques using internal memory cells. The efficiency of these statedsaving techniques depends on the selection of premises for storing the joins. In this paper we first present a general program transformation technique for saving joins which heuristically reorders subgoals and performs predicate splitting optimization, an extension of unfolding. This statedsaving transformation can be applied for any goalddirected and modeldgeneration evaluation. It does not require any information about the query. We show that the program transformation approach is equivalent to the specialized statedsaving implementations known from production systems and modeldgeneration theorem provers. To use the efficient and complete bottomdup evaluation also for query answering, we improve the supplementary extensions of Generalized Magic Sets and Magic Templates rewriting strategies by the presented statedsaving transformation.

An analysis of magic sets and related optimization strategies for logic queries

We analyze the optimization effect of the “magic sets” rewriting technique for datalog queries and present some supplementary or alternative techniques that avoid many shortcomings of the basic technique. Given a magic sets rewritten query, the set of facts generated for the original, nonmagic predicates by the seminaive bottom-up evaluation is characterized precisely. It is shown that—because of the additional magic facts—magic sets processing may result in generating an order of magnitude more facts than the straightforward naive evaluation. A refinement of magic sets in factorized magic sets is defined. These magic sets retain most of the efficiency of original magic sets in regards to the number of nonmagic facts generated and have the property that a linear-time bound with respect to seminaive evaluation is guaranteed in all cases. An alternative technique for magic sets, called envelopes , which has several desirable properties over magic sets, is introduced. Envelope predicates are never recursive with the original predicates; thus, envelopes can be computed as a preprocessing task. Envelopes also allow the utilization of multiple sideways information passing strategies (sips) for a rule. An envelope-transformed program may be “readorned” according to another choice of sips and reoptimized by magic sets (or envelopes), thus making possible an optimization effect that cannot be achieved by magic sets based on a particular choice of sips.

Querying datalog with arrays: Design and implementation issues

This paper deals with the implementation of logic queries where array structures are manipulated. Both top-down and bottom-up implementations of the presented logic language, called Datalog A , are considered. Indeed, SLD-resolution is generalized to realize Datalog A top-down query answering. Further, a fixpoint based evaluation of Datalog A queries is introduced, which forms the basis for efficient bottom-up implementation of queries obtained by generalizing rewriting techniques such as magic set method to the case of Datalog A programs.

Cost-based optimization for magic

Magic sets rewriting is a well-known optimization heuristic for complex decision-support queries. There can be many variants of this rewriting even for a single query, which differ greatly in execution performance. We propose cost-based techniques for selecting an efficient variant from the many choices.Our first contribution is a practical scheme that models magic sets rewriting as a special join method that can be added to any cost-based query optimizer. We derive cost formulas that allow an optimizer to choose the best variant of the rewriting and to decide whether it is beneficial. The order of complexity of the optimization process is preserved by limiting the search space in a reasonable manner. We have implemented this technique in IBM's DB2 C/S V2 database system. Our performance measurements demonstrate that the cost-based magic optimization technique performs well, and that without it, several poor decisions could be made.Our second contribution is a formal algebraic model of magic sets rewriting, based on an extension of the multiset relational algebra, which cleanly defines the search space and can be used in a rule-based optimizer. We introduce the multiset θ -semijoin operator, and derive equivalence rules involving this operator. We demonstrate that magic sets rewriting for non-recursive SQL queries can be modeled as a sequential composition of these equivalence rules.

A Framework for Testing Safety and Effective Computability

This paper presents a methodology for testing a logic program containing function symbols and built-in predicates forsafetyandeffective computability. Safety is the property that the set of answers for a given query is finite. A related issue is whether the evaluation strategy can effectively compute all answers and terminate. We consider these problems under the assumption that queries are evaluated using a fair bottom-up fixpoint computation. We also model the use of function symbols, to construct complex terms such as lists, and arithmetic operators, by considerating Datalog programs with infinite base relations over whichfiniteness constraintsandmonotonicity constraintsare imposed. One of the main results of this paper is a recursive algorithm,check–clique, to test the safety and effective computability of predicates in arbitrarily complex cliques in the predicate connection graph. This algorithm takes certain procedures as parameters, and its applicability can be strengthened by making these procedures more sophisticated. We specify the properties required of these procedures precisely, and present a formal proof of correctness for the algorithmcheck–clique. This work can be seen as providing a framework for testing safety and effective computability of recursive programs, in some ways analogous to thecapture rulesframework of Ullman. A second important contribution is a framework for analyzing programs that are produced by theMagic Setstransformation utilizingcheck–cliqueto analyze recursive cliques. The transformed program unfortunately often has a clique structure that combines several cliques of the original program. Given the complexity of algorithmcheck–clique, it is important to keep cliques as small as possible. We deal with this problem by considering cliques in an intermediate program, called theadorned program, produced by the Magic Sets transformation. The clique structure of the adorned program is similar to that of the original program, and by showing how to analyze the transformed program in terms of the cliques in the adorned program, we avoid the potentially expensive analysis of the cliques in the transformed program.

On the Expected Size of Recursive Datalog Queries

We present asymptotically exact expressions for the expected sizes of relations defined by three well-studied Datalog recursions, namely the "transitive closure," "same generation," and "canonical factorable recursion." We consider the size of the fixpoints of the recursively defined relations in the above programs, as well as the size of the fixpoints of the relations defined by the rewritten programs generated by the Magic Sets and Factoring rewriting algorithms in response to selection queries. Our results show that even over relatively sparse base relations, the fixpoints of the recursively defined relations are within a small constant factor of their worst-case size bounds, and that the Magic Sets rewriting algorithm on the average produces relations whose fixpoints are within a small constant factor of the corresponding bounds for the recursion without rewriting. The expected size of the fixpoint of the relations produced by the Factoring algorithm, when it applies, is significantly smaller than the expected size of the fixpoints of the relations produced by Magic Sets. This lends credence to the belief that reducing the arity of the recursive predicate is probably more important than restricting the recursion to relevant tuples.

Depth-bounded bottom-up evaluation of logic programs

We present here a depth-bounded bottom-up evaluation algorithm for logic programs. We show that it is sound, complete, and terminating for finite-answer queries if the programs are syntactically restricted to Datalog nS , a class of logic programs with limited function symbols. Datalog nS is an extension of Datalog capable of representing infinite phenomena. Predicates in Datalog nS can have arbitrary unary and limited n-ary function symbols in one distinguished argument. We precisely characterize the computational complexity of depth-bounded evaluation for Datalog nS and compare depth-bounded evaluation with other evaluation methods, top-down and Magic Sets among others. We also show that universal safety (finiteness of query answers for any database) is decidable for Datalog nS .

Argument reduction by factoring

We identify a useful property of a program with respect to a predicate, calledfactoring. While we prove that detecting factorability is undecidable in general, we show that for a large class of programs, the program obtained by applying the Magic Sets transformation is factorable with respect to the recursive predicate. When the factoring property holds, a simple optimization of the program generated by the Magic Sets transformation results in a new program that is never less efficient than the Magic Sets program and is often dramatically more efficient, due to the reduction of the arity of the recursive predicate. We show that the concept of factoring generalizes some previously identified special cases of recursions, including separable recursions and right- and left-linear recursions.

Bottom-up evaluation and query optimization of well-founded models

We present a bottom-up operational procedure for computing well-founded models of allowed programs with negation. This procedure provides a practical method of handling programs that involve unstratified negation in a manner that may be mixed with other evaluation approaches, such as semi-naive evaluation and various program transformations. We define classes of programs and sideways information of passing strategies (sips) for which the magic sets transformation preserves well-founded models with respect to the query. The classes of programs and sips we consider strictly subsume those already considered in the literature, and include stratified programs (with any choice of sips), left-to-right modularly stratified programs (with left-to-right sips) and arbitrary programs (with well-founded sips). For these programs and sips, our procedure for computing well-founded models is applicable to the rewritten programs, thus allowing increased efficiency of specializing a program for a query. Finally, we describe an extension of our bottom-up operational procedure that operates on the magic sets transformation of arbitrary programs with arbitrary sips, and computes query answers with respect to the well-founded models of the original program.

Abstract interpretation: A kind of magic

Magic sets and, more recently, magic templates have been used in the field of deductive databases to facilitate efficient bottom-up evaluation of database queries. Roughly speaking a top-down computation of a definite logic program is simulated by first transforming the program and then executing the new program bottom-up. In this paper we give a new and very simple proof that this approach is equivalent to the collecting interpretation of the abstract interpretation framework for logic programs of Mellish. As a side-effect we are also able to show that “bottom-up” abstract interpretation based on the magic templates transformation is equally powerful as Mellish's abstract interpretation framework, but less powerful than other (more precise) abstract interpretation frameworks.

Chain-split evaluation in deductive databases

Many popularly studied recursions in deductive databases can be compiled into one or a set of highly regular chain generating paths, each of which consists of one or a set of connected predicates. Previous studies on chain-based query evaluation in deductive databases take a chain generating path as an inseparable unit in the evaluation. However, some recursions, especially many functional recursions whose compiled chain consists of infinitely evaluable function(s), should be evaluated by chain-split evaluation, which splits a chain generating path into two portions in the evaluation: an immediately evaluable portion and a delayed-evaluation portion. The necessity of chain-split evaluation is examined from the points of view of both efficiency and finite evaluation, and three chain-split evaluation techniques: magic sets, buffered evaluation, and partial evaluation are developed. Our study shows that chain-split evaluation is a primitive recursive query evaluation technique for different kinds of recursions, and it can be implemented efficiently in deductive databases by extensions to the existing recursive query evaluation methods. >

Modular stratification and magic sets for Datalog programs with negation

A class of “modularly stratified” logic programs is defined. Modular stratification generalizes stratification and local stratification, while allowing programs that are not expressible as stratified programs. For modularly stratified programs, the well-founded semantics coincides with the stable model semantics and makes every ground literal true or false. Modularly stratified programs are weakly stratified, but the converse is false. Unlike some weakly stratified programs, modularly stratified programs can be evaluated in a subgoal-at-a time fashion. An extension of top-down methods with memoing that handles this broader class of programs is presented. A technique for rewriting a modularly stratified program for bottom-up evaluation is demonstrated and extended to include magic-set techniques. The rewritten program, when evaluated bottom-up, gives correct answers according to the well-founded semantics, but much more efficiently than computing the complete well-founded model. A one-to-one correspondence between steps of the extended top-down method and steps during the bottom-up evaluation of the magic-rewritten program is exhibited, demonstrating that the complexity of the two methods is the same. Extensions of modular stratification to other operators such as set-grouping and aggregation, which have traditionally been stratified to prevent semantic difficulties, are discussed.

The Glue-Nail deductive database system: Design, implementation, and evaluation

We describe the design and implementation of the Glue-Nail deductive database system. Nail is a purely declarative query language; Glue is a procedural language used for non-query activities. The two languages combined are sufficient to write a complete application. Nail and Glue code are both compiled into the target language IGlue. The Nail compiler uses variants of the magic sets algorithm and supports well-founded models. The Glue compiler's static optimizer uses peephole techniques and data flow analysis to improve code. The IGlue interpreter features a run-time adaptive optimizer that reoptimizes queries and automatically selects indexes. We also describe the Glue-Nail benchmark suite, a set of applications developed to evaluate the Glue-Nail language and to measure the performance of the system.

Bottom-up abstract interpretation of logic programs

This paper presents a formal framework for the bottom-up abstract interpretation of logic programs which can be applied to approximate answer substitutions, partial answer substitutions and call patterns for a given program and arbitrary initial goal. The framework is based on a T p -like semantics defined over a Herbrand universe with variables which has previously been shown to determine the answer substitutions for arbitrary initial goals. The first part of the paper reconstructs this semantics to provide a more adequate initial goals. The first part of the paper reconstructs this semantics to provide a more adequate basis for abstract interpretation. A notion of abstract substitution is introduced and shown to determine an abstract semantic function which for a given program can be applied to approximate the answer substitutions for an arbitrary initial goal. The second part of the paper extends the bottom-up approach to provide approximations of both partial answer substitutions and call patterns. This is achieved by applying Magic Sets and other existing techniques to transform a program in such a way that the answer substitutions of the transformed program correspond to the partial answer substitutions and call patterns of the original program. This facilitates the analysis of concurrent logic programs (ignoring synchronization) and provides a collecting semantics which characterizes both success and call patterns.

Incorporating top-down information into bottom-up hypothetical reasoning

Both bottom-up and top-down approaches have been presented for hypothetical reasoning. However, both have merits and demerits, which are complementary. Thus, hypothetical reasoners combining those approaches are promising. Here, we concern about a bottom-up hypothetical reasoner incorporating top-down information. In order to simulate top-down reasoning on bottom-up reasoners, we can apply the upside-down meta-interpretation method, which is similar to Magic Set and Alexander methods, by transforming a set of Horn clauses into a program incorporating goal information. Unfortunately, it does not achieve speedups for bottom-up hypothetical reasoning because checking consistencies of solutions by negative clauses should be globally evaluated. This paper presents a new method to reduce the consistency checking cost for bottom-up hypothetical reasoning based on the upside-down meta-interpretation. In the transformation algorithm, logical dependencies between a goal and negative clauses are analyzed to find irrelevant negative clauses, so that bottom-up hypothetical reasoning based on the upside-down meta-interpretation can restrict consistency checking of negative clauses to those relevant clauses.

Design and implementation of the glue-nail database system

We describe the design and implementation of the Glue-Nail database system. The Nail language is a purely declarative query language; Glue is a procedural language used for non-query activities. The two languages combined are sufficient to write a complete application. Nail and Glue code both compile into the target language IGlue. The Nail compiler uses variants of the magic sets algorithm, and supports well-founded models. Static optimization is performed by the Glue compiler using techniques that include peephole methods and data flow analysis. The IGlue code is executed by the IGlue interpreter, which features a run-time adaptive optimizer. The three optimizers each deal with separate optimization domains, and experiments indicate that an effective synergism is achieved. The Glue-Nail system is largely complete and has been tested using a suite of representative applications.

Efficient Evaluation of Generalized Left-linear Recursive Queries With Multivalue Binding Sets

The left-linear transformation presented by Ullman, Naughton el al. is a rule-rewriting algorithm, which is similar to the magic sets transformation. Since left-linear recursions are among the most common recursions in practice and the transformed rules can be evaluated bottom-up efficiently, the left-linear transformation has been chosen as one of the best strategies for recursive query processing by the NALL! system at Stanford and the LDL system at MCC. However, there are two strick restraints for the left-linear transformation