Recursive Queries Research Articles

The Complexity of Why-Provenance for Datalog Queries

Datalog is a powerful rule-based language that allows us to express complex recursive queries and has found numerous applications over the years. Explaining why a result to a Datalog query is obtained is an essential task towards explainable and transparent data-intensive applications that rely on Datalog. A standard way of explaining a query result is the so-called why-provenance, which provides information about the witnesses to a query result in the form of subsets of the input database that as a whole can be used to derive that result. To our surprise, despite the fact that the notion of why-provenance for Datalog queries has been around for decades and intensively studied, its computational complexity remains unexplored. Our goal is to fill this gap in the why-provenance literature. Towards this end, we pinpoint the data complexity of why-provenance for Datalog queries and key subclasses thereof. The takeaway of our work is that why-provenance for recursive queries, even if the recursion is limited to be linear, is an intractable problem, whereas for non-recursive queries is highly tractable.

INTERFACES OF VIRTUAL DATA STORAGE IN THE CONDITIONS OF A FAST-CHANGING INFORMATION ENVIRONMENT

The article analyzes data storage architectures, explores the main approaches to cloud technologies, and highlights the features of virtual data storage architecture formation. In the process of analysis, positive and negative features of individual data storage architectures were highlighted. Cloud information technologies are considered from the point of view of their use for data storage and distribution. When considering cloud data storage, attention is focused on their architectural features from the point of view of efficiency and availability for users. Storage efficiency is an important characteristic of cloud storage infrastructure, especially given its emphasis on overall cost savings. Performance has many aspects, but the main task of a cloud storage system is to move data between the user and the remote cloud service provider. The system architecture was designed based on the proposed and modeled methods of organizing access to the cloud storage. According to the designed architecture, the data flows in the system were analyzed and investigated. The developed algorithm for the collection and integration of disparate data based on virtual data warehouses, in contrast to the Inmon and Kimball models, makes it possible to process NoSQL temporal (InfluxDB), graph (Neo4j) and document-oriented (MongoDB) data with a significantly lower value of capacity complexity and the possibility execution of recursive queries. The developed interface should be able to work with relational, temporal, graph and document-oriented data sources. To do this, it must provide the ability to perform SQL-like queries against relational databases, as well as specialized queries for each type of non-relational data. It allows to increase the efficiency of process0ng both relational and non-relational data, which makes it a powerful tool for analyzing and processing information from various sources. The development of virtual data warehouses may be associated with the introduction of new technologies, such as artificial intelligence, blockchain, quantum computing, etc. In addition, further improvements in metadata management and data security are expected.

Convergence of datalog over (Pre-) Semirings

Recursive queries have been traditionally studied in the framework of datalog, a language that restricts recursion to monotone queries over sets, which is guaranteed to converge in polynomial time in the size of the input. But modern big data systems require recursive computations beyond the Boolean space. In this article, we study the convergence of datalog when it is interpreted over an arbitrary semiring. We consider an ordered semiring, define the semantics of a datalog program as a least fixpoint in this semiring, and study the number of steps required to reach that fixpoint, if ever. We identify algebraic properties of the semiring that correspond to certain convergence properties of datalog programs. Finally, we describe a class of ordered semirings on which one can use the semi-naïve evaluation algorithm on any datalog program.

Optimizing Nested Recursive Queries

Datalog is a declarative programming language that has gained popularity in various domains due to its simplicity, expressiveness, and efficiency. But "pure" Datalog is limited to monotone queries, and cannot be used in most practical applications. For that reason, newer systems are relaxing the language by allowing non-monotone queries to be freely combined with recursion. But by departing from the elegant fixpoint semantics of pure datalog, these systems often result in inefficient query execution, for example they perform redundant computations, or use redundant storage. In this paper, we propose Temporel, a system that allows recursion to be freely combined with non-monotone operators. Temporel optimizes the program by compiling it into a novel intermediate representation that we call TempoDL. Our experimental results show that our system outperforms a state-of-the-art Datalog engine as well as a vectorized and a compiled in-memory database system for a wide range of applications from machine learning to graph processing.

Convergence of Datalog over (Pre-) Semirings

Recursive queries have been traditionally studied in the framework of datalog, a language that restricts recursion to monotone queries over sets, which is guaranteed to converge in polynomial time in the size of the input. But modern big data systems require recursive computations beyond the Boolean space. In this paper we study the convergence of datalog when it is interpreted over an arbitrary semiring. We consider an ordered semiring, define the semantics of a datalog program as a least fixpoint in this semiring, and study the number of steps required to reach that fixpoint, if ever. We identify algebraic properties of the semiring that correspond to certain convergence properties of datalog programs. Finally, we describe a class of ordered semirings on which one can generalize the semi-na¨ve evaluation algorithm to compute their minimal fixpoints.

On Monotonic Determinacy and Rewritability for Recursive Queries and Views

A query Q is monotonically determined over a set of views V if Q can be expressed as a monotonic function of the view image. In the case of relational algebra views and queries, monotonic determinacy coincides with rewritability as a union of conjunctive queries, and it is decidable in important special cases, such as for conjunctive query views and queries. We investigate the situation for views and queries in the recursive query language Datalog. We give both positive and negative results about the ability to decide monotonic determinacy, and also about the co-incidence of monotonic determinacy with Datalog rewritability.

Optimizing differentially-maintained recursive queries on dynamic graphs

Differential computation (DC) is a highly general incremental computation/view maintenance technique that can maintain the output of an arbitrary and possibly recursive dataflow computation upon changes to its base inputs. As such, it is a promising technique for graph database management systems (GDBMS) that support continuous recursive queries over dynamic graphs. Although differential computation can be highly efficient for maintaining these queries, it can require prohibitively large amount of memory. This paper studies how to reduce the memory overhead of DC with the goal of increasing the scalability of systems that adopt it. We propose a suite of optimizations that are based on dropping the differences of operators, both completely or partially, and recomputing these differences when necessary. We propose deterministic and probabilistic data structures to keep track of the dropped differences. Extensive experiments demonstrate that the optimizations can improve the scalability of a DC-based continuous query processor.

PRESERVATION OF HIERARCHY STRUCTURES IN RELATIVE DATABASES

Relational database management systems and the SQL language itself do not have any built-in mechanisms for storing and managing hierarchical structures. There are several different ways to represent trees in relational databases. This paper considers the method of modeling hierarchical data structures in the form of Adjacency Lists and Closure Tables. For each method, there are examples of writing queries to solve typical problems encountered when working with tree structures: finding all descendant leaves, all descendants and ancestors of a given leaf, moving a leaf to another ancestor leaf, and deleting leaves with all its descendants. The possibility of using recursive queries when displaying the entire tree in the Adjacency List model is considered. If the depth of the tree is not known, or it is not known at what level the specified element is, the query can not be built by standard means of the SELECT statement, then you need to create a recursive procedure, or write a recursive query. In order to avoid recursion when outputting the whole tree, all nodes of the subtree, and finding the path from a certain place to the root, the modeling of hierarchical data structures is performed in the form of a connection table (Closure Table). This complicates the process of adding a new leaf and moving the leaf to another ancestor leaf. In this case, to simplify the writing of queries, it is suggested to create triggers that will build or rebuild the links. Given the fact that sometimes there is a need to preserve dependent, in particular hierarchical structures in a relational database, you need to be able to plow the model of preservation of such data. The choice of method for solving a specific problem is influenced by the speed of basic operations with trees. Exploring different options for organizing SQL tree structures will allow you to understand and choose the best way to build such a structure in a relational database for a specific task. All SQL queries in this paper were created and tested for Oracle relational databases.

Query Rewriting for Horn-SHIQ Plus Rules

Query answering over Description Logic (DL) ontologies has become a vibrant field of research. Efficient realizations often exploit database technology and rewrite a given query to an equivalent SQL or Datalog query over a database associated with the ontology. This approach has been intensively studied for conjunctive query answering in the DL-Lite and EL families, but is much less explored for more expressive DLs and queries. We present a rewriting-based algorithm for conjunctive query answering over Horn-SHIQ ontologies, possibly extended with recursive rules under limited recursion as in DL+log. This setting not only subsumes both DL-Lite and EL, but also yields an algorithm for answering (limited) recursive queries over Horn-SHIQ ontologies (an undecidable problem for full recursive queries). A prototype implementation shows its potential for applications, as experiments exhibit efficient query answering over full Horn-SHIQ ontologies and benign downscaling to DL-Lite, where it is competitive with comparable state of the art systems.

Temel Parametreleri ve Algoritmalarıyla Bir Gıda İzlenebilirliği Veritabanı Modeli

This study is offering a common food traceability algorithm and a sample database model usable for food related companies. Monitoring food items and related sub-items is important in order to recall them when a food problem occurs and to investigate the source of the problem. If food companies have a common standard structure for traceability, it will be easier to share and transmit these records. In addition, food workers, supervisors and consumers will be able to monitor food more safely and comfortably. For these reasons, a sample structure, considering basic parameters and methods used in food traceability studies, is tried to be established within this study. For this purpose, the basic parameters and relationships required to ensure food traceability in food traceability studies are investigated by content analysis and an entity relationship diagram is created using these parameters. Then an algorithm is prepared to combine the records of food items systematically. Afterwards, the validity of the database is examined through the data available in a food company. In this way, this study aims to make forward and backward food traceability in a simple manner using recursive queries and to provide infrastructure for future standard applications.

Object traceability graph: Applying temporal graph traversals for efficient object traceability

EPC Information Services (EPCIS) is a de-facto standard for information systems for object traceability. Expert and intelligent systems complying with the standard reap benefits from its global interoperability, and thus every application can easily retrieve track and trace information of everyday-objects in an identical manner. However, existing systems are bound to go through scalability issues due to inevitable recursive queries because users are required to handle multiple transformation and aggregation of the objects instead of the systems. In the article, we propose an enhanced EPCIS system called Object Traceability Graph (OTG). The system applies a technique, temporal graph traversal, to resolve the issues. With the system, applications do not need to request recursive queries on their side. Instead, applications are able to represent their ad-hoc traceability queries in a single statement provided by the system. Then, the statement is interpreted and efficiently processed on the system side. In our evaluation, it is shown that our approach enhances the scalability of EPCIS systems by reducing the number of queries and the amount of data transmission. The proposed approach can be applied to existing expert and intelligent systems. Furthermore, we believe that an additional interface, abstracting our approach, is general enough to be included in the standard.

Recursion in SPARQL

The need for recursive queries in the Semantic Web setting is becoming more and more apparent with the emergence of datasets where different pieces of information are connected by complicated patterns. This was acknowledged by the W3C committee by the inclusion of property paths in the SPARQL standard. However, as more data becomes available, it is becoming clear that property paths alone are not enough to capture all recursive queries that the users are interested in, and the literature has already proposed several extensions to allow searching for more complex patterns. We propose a rather different, but simpler approach: add a general purpose recursion operator directly to SPARQL. In this paper we provide a formal syntax and semantics for this proposal, study its theoretical properties, and develop algorithms for evaluating it in practical scenarios. We also show how to implement this extension as a plug-in on top of existing systems, and test its performance on several synthetic and real world datasets, ranging from small graphs, up to the entire Wikidata database.

Blockzone: A Decentralized and Trustworthy Data Plane for DNS

The domain name system (DNS) provides a mapping service between memorable names and numerical internet protocol addresses, and it is a critical infrastructure of the Internet. The authenticity of DNS resolution results is crucial for ensuring the accessibility of Internet services. Hundreds of supplementary specifications of protocols have been proposed to compensate for the security flaws of DNS. However, DNS security incidents still occur frequently. Although DNS is a distributed system, for a specified domain name, only authorized authoritative servers can resolve it. Other servers must obtain the resolution result through a recursive or iterative resolving procedure, which renders DNS vulnerable to various attacks, such as DNS cache poisoning and distributed denial of service (DDoS) attacks. This paper proposes a novel decentralized architecture for a DNS data plane, which is called Blockzone. First, Blockzone utilizes novel mechanisms, which include on-chain authorization and off-chain storage, to implement a decentralized and trustworthy DNS data plane. Second, in contrast to the hierarchical authentication and recursive query of traditional DNS, Blockzone implements a decentralized operation model. This model significantly increases the efficiency of domain name resolution and verification and enhances the security of DNS against DDoS and cache poisoning attacks. In addition, Blockzone is fully compatible with the traditional DNS implementation and can be incrementally deployed as a plug-in service of DNS without changing the DNS protocol or system architecture. The Blockzone scheme can also be generalized to address security issues in other areas, such as the Internet of things and edge computing.

Distribution Policies for Datalog

Modern data management systems extensively use parallelism to speed up query processing over massive volumes of data. This trend has inspired a rich line of research on how to formally reason about the parallel complexity of join computation. In this paper, we go beyond joins and study the parallel evaluation of recursive queries. We introduce a novel framework to reason about multi-round evaluation of Datalog programs, which combines implicit predicate restriction with distribution policies to allow expressing a combination of data-parallel and query-parallel evaluation strategies. Using our framework, we reason about key properties of distributed Datalog evaluation, including parallel-correctness of the evaluation strategy, disjointness of the computation effort, and bounds on the number of communication rounds.

Technical Perspective for

The main processing paradigm in data management is bulk processing. As introduced by Codd in the early 70's, under this paradigm relations are processed in bulk, one operator at a time. When applied to relations, this paradigm leads to relational algebra, and its variants, relational calculus, and SQL. Over the years, data management was faced with the challenge of extending bulk processing operators to new kinds of data, and/or new kinds of queries: nested relations, semistructured data, recursive queries. Each such extension requires significant systems development, which should be accompanied, in fact preceded, by a careful study of the expressive power of the new language. Is it as expressive, more expressive, or less expressive than relational algebra? The answer to this question has profound implications on the ability of data processing engines to optimize, compute, distribute, reuse queries in that language. For example, extending relational algebra with nested relations does not increase its expressive power, while extending it with fixpoint does, explaining why modern query engines have an easier time supporting JSON than recursion.

A solution for synchronous incremental maintenance of materialized views based on SQL recursive query

Materialized views are excessively stored query execution results in the database. They can be used to partially or completely answer queries which will be further appeared instead of re-executing query from the scratch. There is a large number of published works that address the maintenance, especially incremental update, of materialized views and query rewriting for using those ones. Some of them support materialized views based on recursive query in datalog language. Although most of datalog queries can be transferred into SQL queries and vise versa but it is not the case for recursive queries. Recursive queries in the data log try to find all possible transitive closures. Recursive queries in SQL (Common Table Expression – CTE) return direct links but not transitive closures. In this paper, we propose efficient methods for incremental update of materialized views based on CTE; and then propose an algorithm for generating source codes in C language for any input SQL recursive queries. The synthesized source codes implement our proposed incremental update algorithms according to inserted/deleted/updated record set in the base tables. This paper focuses mainly on the recursive queries whose execution results are directed tree-structured data. The two cases of tree node are considered. In the first case, a child node has only one parent node and in the second case, a child node can have many parent nodes. Those two cases represent the two types of relationships between entities in real world, that are one–to–many and many–to–many, respectively. For the one–to–many relationships, the relationship data is accompanied with the records describing the child using some fields. Those fields are set as null in deleting a concrete relationship. For the many–to–many relationships, it is stored in a separate table and the concrete relationships are removed by deleting describing records from that table. Considering of enforcing referential integrity may help to reduce the searching space and therefore, help to improve the performance. However, the set of tree nodes or tree edges can be manipulated. All those combinations lead to different algorithms. The experimental results are provided and discussed to confirm the effectiveness of our proposed methods

A Case for Stale Synchronous Distributed Model for Declarative Recursive Computation

AbstractA large class of traditional graph and data mining algorithms can be concisely expressed in Datalog, and other Logic-based languages, once aggregates are allowed in recursion. In fact, for most BigData algorithms, the difficult semantic issues raised by the use of non-monotonic aggregates in recursion are solved byPre-Mappability(${\cal P}$reM), a property that assures that for a program with aggregates in recursion there is an equivalent aggregate-stratified program. In this paper we show that, by bringing together the formal abstract semantics of stratified programs with the efficient operational one of unstratified programs,$\[{\cal P}\]$reMcan also facilitate and improve their parallel execution. We prove that$\[{\cal P}\]$reM-optimized lock-free and decomposable parallel semi-naive evaluations produce the same results as the single executor programs. Therefore,$\[{\cal P}\]$reMcan be assimilated into the data-parallel computation plans of different distributed systems, irrespective of whether these follow bulk synchronous parallel (BSP) or asynchronous computing models. In addition, we show that non-linear recursive queries can be evaluated using a hybrid stale synchronous parallel (SSP) model on distributed environments. After providing a formal correctness proof for the recursive query evaluation with$\[{\cal P}\]$reMunder this relaxed synchronization model, we present experimental evidence of its benefits.

Scaling-up in-memory datalog processing

Recursive query processing has experienced a recent resurgence, as a result of its use in many modern application domains, including data integration, graph analytics, security, program analysis, networking and decision making. Due to the large volumes of data being processed, several research efforts across multiple communities have explored how to scale up recursive queries, typically expressed in Datalog. Our experience with these tools indicate that their performance does not translate across domains---e.g., a tool designed for large-scale graph analytics does not exhibit the same performance on program-analysis tasks, and vice versa. Starting from the above observation, we make the following two contributions. First, we perform a detailed experimental evaluation comparing a number of state-of-the-art Datalog systems on a wide spectrum of graph analytics and program-analysis tasks, and summarize the pros and cons of existing techniques. Second, we design and implement our own general-purpose Datalog engine, called RecStep, on top of a parallel single-node relational system. We outline the techniques we applied on RecStep, as well as the contribution of each technique to the overall performance. Using RecStep as a baseline, we demonstrate that it generally out-performs state-of-the-art parallel Datalog engines on complex and large-scale Datalog evaluation, by a 4-6X margin. An additional insight from our work is that it is possible to build a high-performance Datalog system on top of a relational engine, an idea that has been dismissed in past work.

SQL-G: Efficient Graph Analytics by SQL

Querying graphs and conducting graph analytics become important in data processing since many real applications are dealing with massive graphs, such as online social networks, Semantic Web, knowledge graphs, etc. Over the years, many distributed graph processing systems have been developed to support graph analytics using various programming models, and many graph querying languages have been proposed. A natural question that arises is how to integrate graph data and traditional non-graph data in a distributed system for users to conduct analytics. There are two issues. One issue is related to expressiveness on how to specify graph analytics as well as data analytics by a querying language. The other issue is related to efficiency on how to process analytics in a distributed system. For the first issue, SQL is a best candidate, since SQL is a well-accepted language for data processing. We concentrate on SQL for graph analytics. Our early work shows that graph analytics can be supported by SQL in a way from “semiring + while” to “relational algebra + while” via the enhanced recursive SQL queries. In this article, we focus on the second issue on how to process such enhanced recursive SQL queries based on the GAS ( Gather - Apply - Scatter ) model under which efficient graph processing systems can be developed. To demonstrate the efficiency, we implemented a system by tightly coupling Spark SQL and GraphX on Spark which is one of the most popular in-memory data-flow processing platforms. First, we enhance Spark SQL by adding the capability of supporting the enhanced recursive SQL queries for graph analytics. In this regard, graph analytics can be processed using a distributed SQL engine alone. Second, we further propose new transformation rules to optimize/translate the operations for recursive SQL queries to the operations by GraphX . In this regard, graph analytics by SQL can be processed in a similar way as done by a distributed graph processing system using the APIs provided by the system. We conduct extensive performance studies to test graph analytics using large real graphs. We show that our approach can achieve similar or even higher efficiency, in comparison to the built-in graph algorithms in the existing graph processing systems.

On Safety of Unary and Non-unary IFP-operators

In this paper, we investigate the safety of unary inflationary fixed point operators (IFPoperators). The safety is a computability in finitely many steps. IFP-operators exactly correspond to recursive SQL-queries hence this problem has a value for database theory. The problem appears from the fact that if recursive queries contain universe functions and relations, then its execution can fall into an infinite loop. Moreover, universal computational devices (Turing machines et al.) can be modelled by such queries. Hence the problem of the finite computability for such queries is undecidable. In our previous works we established some properties of a universe which imply the finite computability of all IFP-operators in the universe. Here, we investigate a connection between an arity of IFP-operators and their safety. We prove that some results for general IFP-operators don’t hold for unary ones. We construct a universe where all unary unnesed IFP-operators are safe. But in this universe there exist unsafe nested unary IFP-operators and unsafe unnested binary IFP-operators. This differs from general IFP-operators because in general case the safety of all unnesed IFP-operators implies the safety of all IFP-operators. Also there exist elementary equivalent universes where some unary unnesed IFPoperators become unsafe. For general IFP-operators it is also impossible.