Datalog Programs Research Articles

Repairing inconsistent dimensions in data warehouses

A dimension in a data warehouse (DW) is a set of elements connected by a hierarchical relationship. The elements are used to view summaries of data at different levels of abstraction. In order to support an efficient processing of such summaries, a dimension is usually required to satisfy different classes of integrity constraints. In scenarios where the constraints properly capture the semantics of the DW data, but they are not satisfied by the dimension, the problem of repairing (correcting) the dimension arises. In this paper, we study the problem of repairing a dimension in the context of two main classes of integrity constraints: strictness and covering constraints. We introduce the notion of minimal repair of a dimension: a new dimension that is consistent with respect to the set of integrity constraints, which is obtained by applying a minimal number of updates to the original dimension. We study the complexity of obtaining minimal repairs, and show how they can be characterized using Datalog programs with weak constraints under the stable model semantics.

Datalog and constraint satisfaction with infinite templates

On finite structures, there is a well-known connection between the expressive power of Datalog, finite variable logics, the existential pebble game, and bounded hypertree duality. We study this connection for infinite structures. This has applications for constraint satisfaction with infinite templates. If the template Γ is ω-categorical, we present various equivalent characterizations of those Γ such that the constraint satisfaction problem (CSP) for Γ can be solved by a Datalog program. We also show that CSP(Γ) can be solved in polynomial time for arbitrary ω-categorical structures Γ if the input is restricted to instances of bounded treewidth. Finally, we characterize those ω-categorical templates whose CSP has Datalog width 1, and those whose CSP has strict Datalog width k.

Outer Joins in a Deductive Database System

Outer joins are extended relational algebra operations intended to deal with unknown information represented with null values. This work shows an approach to embed both null values and outer join operations in the deductive database system DES (Datalog Educational System), which uses Datalog as a query language. This system also supports SQL, where views and queries are compiled to Datalog programs. So, as SQL statements are ultimately solved by a Datalog engine, it became a need to integrate null-related operations into Datalog in order to support a wider set of SQL. Since DES implements a top-down-driven bottom-up stratified fixpoint computation based on tabling for solving Datalog queries, we show how to compute outer joins in such a context by means of source-to-source transformations applied to Datalog programs.

Magic Sets for disjunctive Datalog programs

In this paper, a new technique for the optimization of (partially) bound queries over disjunctive Datalog programs with stratified negation is presented. The technique exploits the propagation of query bindings and extends the Magic Set optimization technique (originally defined for non-disjunctive programs).An important feature of disjunctive Datalog programs is non-monotonicity, which calls for non-deterministic implementations, such as backtracking search. A distinguishing characteristic of the new method is that the optimization can be exploited also during the non-deterministic phase. In particular, after some assumptions have been made during the computation, parts of the program may become irrelevant to a query under these assumptions. This allows for dynamic pruning of the search space. In contrast, the effect of the previously defined Magic Set methods for disjunctive Datalog is limited to the deterministic portion of the process. In this way, the potential performance gain by using the proposed method can be exponential, as could be observed empirically.The correctness of the method is established and proved in a formal way thanks to a strong relationship between Magic Sets and unfounded sets that has not been studied in the literature before. This knowledge allows for extending the method and the correctness proof also to programs with stratified negation in a natural way.The proposed method has been implemented in the DLV system and various experiments on synthetic as well as on real-world data have been conducted. The experimental results on synthetic data confirm the utility of Magic Sets for disjunctive Datalog, and they highlight the computational gain that may be obtained by the new method with respect to the previously proposed Magic Set method for disjunctive Datalog programs. Further experiments on data taken from a real-life application show the benefits of the Magic Set method within an application scenario that has received considerable attention in recent years, the problem of answering user queries over possibly inconsistent databases originating from integration of autonomous sources of information.

Using Logic Programming to Represent Information Content Inclusion Relations

Datalog is a widely recognised language for a certain class of deductive databases. Information Content Inclusion Relation (IIR) formulates a general, information theoretic relationship between: data constructs; between data constructs and real world objects, and between real world objects. IIR is particularly concerned with the information that data carry. It would therefore seem desirable to find out whether IIR and reasoning based on IIR may be implemented by using 'safe' Datalog. We present and prove the following theorem: Any database system that can be modelled using IIR can be represented as a 'safe' Datalog program.} This paper explores the nature of the relationship between the two frameworks for representing domains of application, in order that such representations of IIR by `safe' Datalog can then be used as a tool for the analysis of any site that can be approached with the notion of information content, and in particular any given database, and hence how a database works may be approached in terms of information content of events.

A runtime approach to model-generic translation of schema and data

To support heterogeneity is a major requirement in current approaches to integration and transformation of data. This paper proposes a new approach to the translation of schema and data from one data model to another, and we illustrate its implementation in the tool MIDST-RT. We leverage on our previous work on MIDST, a platform conceived to perform translations in an off-line fashion. In such an approach, the source database (both schema and data) is imported into a repository, where it is stored in a universal model. Then, the translation is applied within the tool as a composition of elementary transformation steps, specified as Datalog programs. Finally, the result (again both schema and data) is exported into the operational system. Here we illustrate a new, lightweight approach where the database is not imported. MIDST-RT needs only to know the schema of the source database and the model of the target one, and generates views on the operational system that expose the underlying data according to the corresponding schema in the target model. Views are generated in an almost automatic way, on the basis of the Datalog rules for schema translation. The proposed solution can be applied to different scenarios, which include data and application migration, data interchange, and object-to-relational mapping between applications and databases.

Survey of directly mapping SQL databases to the Semantic Web

AbstractThe Semantic Web anticipates integrated access to a large number of information sources on the Internet represented as Resource Description Framework (RDF). Given the large number of websites that are backed by SQL databases, methods that automate the translation of those databases to RDF are crucial. One approach, taken by a number of researchers, is to directly map the SQL schema to an equivalent Web Ontology Language (OWL) or RDF Schema representation, which in turn, implies an RDF representation for the relational data. This paper reviews this research, and derives a consolidated, overarching set of translation rules expressible as a stratified Datalog program. We present all the possible key combinations in an SQL schema and consider their implied semantic properties. We review the approaches and characterize them with respect to the scope of their coverage of SQL constructs.

Building a consensus: A rectangle covering problem

The other day, over a very pleasant lunch in the restaurant of Oxford's recently renovated Ashmolean Museum, Oege de Moor gave me a problem about rectangles. The problem is explained more fully later, but roughly speaking one is given a finite set of rectangles RS and a rectangle R completely covered by RS. The task is to construct a single rectangle covering R among the elements of a larger set of rectangles associated with RS, called the saturation of RS. The saturation of RS is the closure of RS under so-called consensus operations, a term coined in (Quine, 1959), in which two rectangles are combined in two distinct ways to form new rectangles. The rectangle problem is a simplified version of containment-checking, a crucial component in a type inference algorithm for Datalog programs (Schäfer & de Moor, 2010). 19 In the Schäfer-de Moor algorithm the problem is generalised to cubes in n-space rather than rectangles in two-space, the components of each cube are given by propositional formulae rather than by intervals on the real line, and certain equality and inhabitation constraints are taken into account. Oege felt that the central proof, Lemma 15 in (Schäfer & de Moor, 2010), deserved to be simplified so he posed the rectangle problem as a special case. This pearl was composed in response to the challenge.

Technical Note: Difficult Lessons Learned in Measuring Stage and Flow Rate on Small Watersheds

Submersible pressure transducers integrated with data loggers are commonly used to measure stage (water level) and estimate flow rate from established stage-discharge relationships; however, proper data quality assurance is important to reduce uncertainty in resulting estimates. For 2 yr, USDA-ARS researchers in Watkinsville, Georgia, continuously monitored flow rate from a spring using a flume fitted with a pressure transducer and data logger and performed periodic manual flow rate checks. Initially, during a 162-day period in 2003 with 72 comparisons, the automated system underestimated the mean flow rate by approximately 17% due to presumed drift and lack of frequent calibration. After calibrating the transducer to adjust for drift with the data logger program, 66 comparisons during a 349-day period in 2004 showed that the difference in mean flow rate between the manual and automated estimates was < 5%. During 235 days late in the monitoring period, the automated system overestimated the mean flow rate by approximately 29% because of another apparent drift. In a similar set-up at a pond outflow, a drift of 32 mm in transducer output caused underestimation of pond outflow by 27% during a 30-day period. Users should not assume that pressure transducer-based automated water level measurement systems continuously record correct data. Routine verification of transducer output is crucial to avoid inaccurate (and possibly useless) stage and flow rate estimates.

Querying Data Sources that Export Infinite Sets of Views

We study the problem of querying data sources that accept only a limited set of queries, such as sources accessible by Web services which can implement very large (potentially infinite) families of queries. We revisit a classical setting in which the application queries are conjunctive queries and the source accepts families of conjunctive queries specified as the expansions of a (potentially recursive) Datalog program with parameters.We say that query Q is expressible by the program \(\ensuremath{\mathcal{P}}\) if it is equivalent to some expansion of \(\ensuremath{\mathcal{P}}\). Q is supported by \(\ensuremath{\mathcal{P}}\) if it has an equivalent rewriting using some finite set of \(\ensuremath{\mathcal{P}}\)’s expansions. We present the first study of expressibility and support for sources that satisfy integrity constraints, which is generally the case in practice.

Monadic datalog over finite structures of bounded treewidth

Bounded treewidth and monadic second-order (MSO) logic have proved to be key concepts in establishing fixed-parameter tractability results. Indeed, by Courcelle's Theorem we know that any property of finite structures, which is expressible by an MSO sentence, can be decided in linear time (data complexity) if the structures have bounded treewidth. In principle, Courcelle's Theorem can be applied directly to construct concrete algorithms by transforming the MSO evaluation problem into a tree language recognition problem. The latter can then be solved via a finite tree automaton (FTA). However, this approach has turned out to be problematical, since even relatively simple MSO formulae may lead to a “state explosion” of the FTA. In this work we propose monadic datalog (i.e., datalog where all intentional predicate symbols are unary) as an alternative method to tackle this class of fixed-parameter tractable problems. We show that if some property of finite structures is expressible in MSO then this property can also be expressed by means of a monadic datalog program over the decomposed structure : we mean by this that the original structure is augmented with new elements and new relations that encode one of its tree decompositions. In the first place, we thus compare the expressive power of two query languages. However, we also show that the resulting fragment of datalog can be evaluated in linear time (both with respect to the program size and with respect to the data size). Hence, our transformation of an MSO query into a monadic datalog program yields an alternative proof of Courcelle's Theorem. In order to actually construct efficient algorithms for problems whose tractability is due to Courcelle's Theorem, we propose to use a fragment of full (i.e., not necessarily monadic) datalog which allows for a succinct representation of the corresponding monadic datalog programs and for an efficient execution. This new approach is put to work by devising datalog programs for the 3-Colorability problem of graphs and for the PRIMALITY problem of relational schemas (i.e., testing if some attribute in a relational schema is part of a key). We also report on experimental results with a prototype implementation.

Query Languages for Data Exchange: Beyond Unions of Conjunctive Queries

The class of unions of conjunctive queries (UCQ) has been shown to be particularly well-behaved for data exchange; its certain answers can be computed in polynomial time (in terms of data complexity). However, this is not the only class with this property; the certain answers to any Datalog program can also can be computed in polynomial time. The problem is that both UCQ and Datalog do not allow negated atoms, as adding an unrestricted form of negation to these languages yields to intractability. In this paper, we propose a language called Datalog C(≠) that extends Datalog with a restricted form of negation, and study some of its fundamental properties. In particular, we show that the certain answers to a Datalog C(≠) program can be computed in polynomial time (in terms of data complexity), and that every union of conjunctive queries with at most one inequality or negated relational atom per disjunct, can be efficiently rewritten as a Datalog C(≠) program in the context of data exchange. Furthermore, we show that this is also the case for a syntactic restriction of the class of unions of conjunctive queries with at most two inequalities per disjunct. This syntactic restriction is given by two conditions that are optimal, in the sense that computing certain answers becomes intractable if one removes any of them. Finally, we provide a thorough analysis of the combined complexity of computing certain answers to Datalog C(≠) programs and other related query languages. In particular, we show that this problem is Exptime-complete for Datalog C(≠), even if one restricts to conjunctive queries with single inequalities, which is a fragment of Datalog C(≠) by the result mentioned above. Furthermore, we show that the combined complexity is coNexptime-complete for the case of conjunctive queries with k inequalities, for every k≥2.

Type inference for datalog with complex type hierarchies

Type inference for Datalog can be understood as the problem of mapping programs to a sublanguage for which containment is decidable. To wit, given a program in Datalog, a schema describing the types of extensional relations, and a user-supplied set of facts about the basic types (stating conditions such as disjointness, implication or equivalence), we aim to infer an over-approximation of the semantics of the program, which should be expressible in a suitable sublanguage of Datalog. We argue that Datalog with monadic extensionals is an appropriate choice for that sublanguage of types, and we present an inference algorithm. The inference algorithm is proved sound, and we also show that it infers the tightest possible over-approximation for a large class of Datalog programs. Furthermore, we present a practical containment check for a large subset of our type language. The crux of that containment check is a novel generalisation of Quine's procedure for computing prime implicants. The type system has been implemented in a state-of-the-art industrial database system, and we report on experiments with this implementation.

Undecidability and intractability results concerning datalog programs and their persistency numbers

The relation between Datalog programs and homomorphism problems, and, between Datalog programs and bounded treewidth structures has been recognized for some time and given much attention recently. Additionally, the essential role of persistent variables (in program expansions) for solving several relevant problems has also started to be observed. In Afrati et al. [2005] the general notion of program persistencies was refined into four notions (two syntactical ones and two semantical ones) and the interrelationship between these four persistency numbers was studied. In the present article (1) we prove undecidability results concerning the semantical notions of persistency number--modulo equivalence, of persistency number and of characteristic integer, (2) we exhibit new classes of programs for which boundedness is undecidable and (3) we prove intractabiltity results concerning the syntactical notions of weak persistency number and of weak characteristic integer.

Strictly declarative specification of sophisticated points-to analyses

We present the DOOP framework for points-to analysis of Java programs. DOOP builds on the idea of specifying pointer analysis algorithms declaratively, using Datalog: a logic-based language for defining (recursive) relations. We carry the declarative approach further than past work by describing the full end-to-end analysis in Datalog and optimizing aggressively using a novel technique specifically targeting highly recursive Datalog programs. As a result, DOOP achieves several benefits, including full order-of-magnitude improvements in runtime. We compare DOOP with Lhotak and Hendren's PADDLE, which defines the state of the art for context-sensitive analyses. For the exact same logical points-to definitions (and, consequently, identical precision) DOOP is more than 15x faster than PADDLE for a 1-call-site sensitive analysis of the DaCapo benchmarks, with lower but still substantial speedups for other important analyses. Additionally, DOOP scales to very precise analyses that are impossible with PADDLE and Whaley et al.'s bddbddb, directly addressing open problems in past literature. Finally, our implementation is modular and can be easily configured to analyses with a wide range of characteristics, largely due to its declarativeness.

On-the-fly progress detection in iterative stream queries

Multiple researchers have proposed cyclic query plans for evaluating iterative queries over streams or rapidly changing input. The Declarative Networking community uses cyclic plans to evaluate Datalog programs that track reachability and other graph traversals on networks. Cyclic query plans can also evaluate pattern-matching and other queries based on event sequences. An issue with cyclic queries over dynamic inputs is knowing when the query result has progressed to a certain point in the input, since the number of iterations is data dependent. One option is a "strictly staged" computation, where the query plan quiesces between inputs. This option introduces significant latency, and may also "underload" inter-operator buffers. An alternative is to settle for soft guarantees, such as "eventual consistency". Such imprecision can make it difficult, for example, to know when to purge state from stateful operators. We propose a third option in which cyclic queries run continuously, but detect progress "on the fly" by means of a Flying Fixed-Point (FFP) operator. FFP sits on the cyclic loop and circulates speculative predictions on forward progress, which it then validates. FFP is always able to track progress for a class of queries we term strongly convergent . A key advantage of FFP is that it works with existing algebra operators, thereby inheriting their capabilities, such as windowing and dealing with out-of-order input. Also, for stream systems that explicitly model input-event lifetimes, we know exactly which values are in the query result at each point in time. A key implementation decision is the method for speculating. Using the high-water mark of data events minimizes the number of speculative punctuations. Probing operators on the cyclic loop to determine their external progress circulates many more speculative messages, but tracks actual output progress more closely. We show how a hybrid approach limits predictions while coming close the progress-tracking ability of Probing.

The Monadic Second-order Logic Evaluation Problem on Finite Colored Trees: a Database-theoretic Approach

We model the monadic second-order logic (MSO) evaluation problem on finite colored trees in a purely database theoretic framework, based on the well-knownMSO-automata connection: we reduce the problem to an acyclic conjunctive query evaluation problem on the one hand and to a monadic datalog evaluation problem on the other hand. This approach offers the possibility to solve the MSO problem using optimized evaluation methods for relational algebra expressions and for datalog programs (such as Yannakakis algorithm [27] and the rewriting method using resolutionbased filtering referred to as magic sets method in [3]): we use these methods for evaluating our queries and giving estimates of their complexity. This is the first time, to our knowledge, that a solution to the MSO evaluation problem related to relational algebra is given; furthermore, thanks to this reduction, we prove that the automata-based algorithm given in [8] constitutes a particular instance of Yannakakis algorithm. Besides the optimized database methods that we propose for solving the MSO evaluation problem, our results prove that MSO-definable queries over colored trees are datalog-definable; this result subsumes the corresponding result in [12] which states that unary MSO queries are monadic datalog-definable and it also subsumes the well-known result that any MSO-definable class of trees is monadic datalog-definable.

A New Proposal for Debugging Datalog Programs

In this paper, we propose to apply declarative debugging to Datalog programs. Our approach relies on program semantics rather than on the computation mechanism. The debugging process starts when the user detects an unexpected answer. By asking questions about the intended semantics, the debugger looks for incorrect program relations. While usual declarative debuggers for other languages are based on computation trees, we show that graphs are more convenient structures for representing Datalog computations. The theoretical framework is complemented by the implementation of a debugger for the deductive database system DES, a publicly available open-source project.

Weighted monadic datalog

Based on monadic datalog, we introduce the concept of weighted monadic datalog over unranked trees. This provides a query language that can be used to extract quantitative information from semi-structured databases where the quantities are taken from some semiring S . We show that weighted monadic datalog is as expressive as weighted tree automata on unranked trees. Moreover, we prove that a query can be evaluated efficiently on an unranked tree provided that (i) S is commutative and the underlying datalog program is non-circular or (ii) S is a finite and commutative ω -cpo semiring.

Reasoning in Description Logics by a Reduction to Disjunctive Datalog

As applications of description logics proliferate, efficient reasoning with knowledge bases containing many assertions becomes ever more important. For such cases, we developed a novel reasoning algorithm that reduces a $\mathcal{SHIQ}$ knowledge base to a disjunctive datalog program while preserving the set of ground consequences. Queries can then be answered in the resulting program while reusing existing and practically proven optimization techniques of deductive databases, such as join-order optimizations or magic sets. Moreover, we use our algorithm to derive precise data complexity bounds: we show that $\mathcal{SHIQ}$ is data complete for NP, and we identify an expressive fragment of $\mathcal{SHIQ}$ with polynomial data complexity.