Inductive Logic Programming Techniques Research Articles

A Critical Review of Inductive Logic Programming Techniques for Explainable AI.

Despite recent advances in modern machine learning algorithms, the opaqueness of their underlying mechanisms continues to be an obstacle in adoption. To instill confidence and trust in artificial intelligence (AI) systems, explainable AI (XAI) has emerged as a response to improve modern machine learning algorithms' explainability. Inductive logic programming (ILP), a subfield of symbolic AI, plays a promising role in generating interpretable explanations because of its intuitive logic-driven framework. ILP effectively leverages abductive reasoning to generate explainable first-order clausal theories from examples and background knowledge. However, several challenges in developing methods inspired by ILP need to be addressed for their successful application in practice. For example, the existing ILP systems often have a vast solution space, and the induced solutions are very sensitive to noises and disturbances. This survey paper summarizes the recent advances in ILP and a discussion of statistical relational learning (SRL) and neural-symbolic algorithms, which offer synergistic views to ILP. Following a critical review of the recent advances, we delineate observed challenges and highlight potential avenues of further ILP-motivated research toward developing self-explanatory AI systems.

Conceptual model visual simulation and the inductive learning of missing domain constraints

Conceptual modeling plays a fundamental role in information systems engineering, and in data and systems interoperability. To play their role as instruments for domain modeling, conceptual models must contain the exact set of constraints that represent the worldview of the relevant domain stakeholders. However, as empirical results show, conceptual modelers are subject to cognitive limitations and biases and, hence, in practice, they systematically produce models that fall short in that respect. Moreover, automating the process of formally assessing conceptual models in this sense (i.e., model validation) is notoriously hard, mainly because the intended worldview at hand lies in the mind of these stakeholders. In this paper, we provide a novel approach to model validation and automated constraint learning that combines, on one hand, Model Finding via the visual simulation of that model’s valid instances and, on the other hand, Inductive Logic Programming techniques. In our approach, we properly channel the results produced by the application of a visual model finding technique as input to a learning process. We then show how the approach is able to support the modeler in identifying missing constraints from the original model. The approach is validated against a catalog of empirically-elicited conceptual modeling anti-patterns. As we show here, the approach is able to support the automated learning of constraints that are needed to rectify a number of relevant anti-patterns in this catalog.

Symbolic AI for XAI: Evaluating LFIT Inductive Programming for Explaining Biases in Machine Learning

Machine learning methods are growing in relevance for biometrics and personal information processing in domains such as forensics, e-health, recruitment, and e-learning. In these domains, white-box (human-readable) explanations of systems built on machine learning methods become crucial. Inductive logic programming (ILP) is a subfield of symbolic AI aimed to automatically learn declarative theories about the processing of data. Learning from interpretation transition (LFIT) is an ILP technique that can learn a propositional logic theory equivalent to a given black-box system (under certain conditions). The present work takes a first step to a general methodology to incorporate accurate declarative explanations to classic machine learning by checking the viability of LFIT in a specific AI application scenario: fair recruitment based on an automatic tool generated with machine learning methods for ranking Curricula Vitae that incorporates soft biometric information (gender and ethnicity). We show the expressiveness of LFIT for this specific problem and propose a scheme that can be applicable to other domains. In order to check the ability to cope with other domains no matter the machine learning paradigm used, we have done a preliminary test of the expressiveness of LFIT, feeding it with a real dataset about adult incomes taken from the US census, in which we consider the income level as a function of the rest of attributes to verify if LFIT can provide logical theory to support and explain to what extent higher incomes are biased by gender and ethnicity.

Learning Higher-Order Programs through Predicate Invention

A key feature of inductive logic programming (ILP) is its ability to learn first-order programs, which are intrinsically more expressive than propositional programs. In this paper, we introduce ILP techniques to learn higher-order programs. We implement our idea in Metagolho, an ILP system which can learn higher-order programs with higher-order predicate invention. Our experiments show that, compared to first-order programs, learning higher-order programs can significantly improve predictive accuracies and reduce learning times.

Interpreting and predicting social commerce intention based on knowledge graph analysis

There have been significant efforts to understand, describe, and predict the social commerce intention of users in the areas of social commerce and web data management. Based on recent developments in knowledge graph and inductive logic programming in artificial intelligence, in this paper, we propose a knowledge-graph-based social commerce intention analysis method. In particular, a knowledge base is constructed to represent the social commerce environment by integrating information related to social relationships, social commerce factors, and domain background knowledge. In this study, knowledge graphs are used to represent and visualize the entities and relationships related to social commerce, while inductive logic programming techniques are used to discover implicit information that can be used to interpret the information behaviors and intentions of the users. Evaluation tests confirmed the effectiveness of the proposed method. In addition, the feasibility of using knowledge graphs and knowledge-based data mining techniques in the social commerce environment is also confirmed.

Learning efficient logic programs

When machine learning programs from data, we ideally want to learn efficient rather than inefficient programs. However, existing inductive logic programming (ILP) techniques cannot distinguish between the efficiencies of programs, such as permutation sort (n!) and merge sort O(n;log;n). To address this limitation, we introduce Metaopt, an ILP system which iteratively learns lower cost logic programs, each time further restricting the hypothesis space. We prove that given sufficiently large numbers of examples, Metaopt converges on minimal cost programs, and our experiments show that in practice only small numbers of examples are needed. To learn minimal time-complexity programs, including non-deterministic programs, we introduce a cost function called tree cost which measures the size of the SLD-tree searched when a program is given a goal. Our experiments on programming puzzles, robot strategies, and real-world string transformation problems show that Metaopt learns minimal cost programs. To our knowledge, Metaopt is the first machine learning approach that, given sufficient numbers of training examples, is guaranteed to learn minimal cost logic programs, including minimal time-complexity programs.

Automatic discovery of concepts and actions

A truly autonomous artificial intelligence agent should be able to drive its own learning process. That is, decide what to explore and what to learn, identifying what constitutes potential useful data as examples of concepts or what strategy to follow to solve a new task. Different efforts have been developed in machine learning towards this aim. Approaches that introduce new concepts, like predicate invention in Inductive Logic Programming (ILP) techniques, normally require the selection of examples by the user. Techniques that learn behavior policies through exploration like Reinforcement Learning (RL) with intrinsic motivation, to guide the agent into interesting areas to discover new goals, assume that all the states and actions are predefined in advance. In this paper, we describe a system, called ADC, that combines techniques from ILP with predicate invention and RL with intrinsic motivation to discover new concepts, states and actions to learn behavior policies. ADC drives its own learning process, collecting its own examples for autonomously learning concepts. These new concepts can be used to describe its environment and define new states and actions used to learn behaviors to solve tasks. We show the effectiveness of our approach in simulated robotics environments.

Modeling violin performances using inductive logic programming

Professional musicians intuitively manipulate sound properties such as pitch, timing, amplitude and timbre in order to produce expressive performances of particular pieces. However, there is little explicit information about how and in which musical contexts this manipulation occurs. In this paper we describe a machine learning approach to modeling the knowledge applied by a musician when performing a score in order to produce an expressive performance. In particular, we apply inductive logic programming techniques in order to automatically learn models for both understanding and generating expressive violin performances.

Computing intensional answers to questions – An inductive logic programming approach

Research on natural language interfaces has mainly concentrated on question interpretation as well as answer computation, but not focused as much on answer presentation. In most natural language interfaces, answers are in fact provided extensionally as a list of all those instances satisfying the query description. In this paper, we aim to go beyond such a mere listing of facts and move towards producing additional descriptions of the query results referred to as intensional answers. We define an intensional answer (IA) as a logical description of the actual set of answer items to a given query in terms of properties that are shared by exactly these answer items. We argue that IAs can enhance a user’s understanding of the answer itself but also of the underlying knowledge base. In particular, we present an approach for computing an intensional answer given an extensional answer (i.e. a set of entities) returned as a result of a question. In our approach, an intensional answer is represented by a clause and computed based on inductive logic programming (ILP) techniques, in particular bottom-up clause generalization. The approach is evaluated in terms of usefulness and time performance, and we discuss its potential for helping to detect flaws in the knowledge base as well as to interactively enrich it with new knowledge. While the approach is used in the context of a natural language question answering system in our settings, it clearly has applications beyond, e.g. in the context of research on generating referring expressions.

Learning rules from multisource data for cardiac monitoring

This paper formalises the concept of learning symbolic rules from multisource data in a cardiac monitoring context. Our sources, electrocardiograms and arterial blood pressure measures, describe cardiac behaviours from different viewpoints. To learn interpretable rules, we use an Inductive Logic Programming (ILP) method. We develop an original strategy to cope with the dimensionality issues caused by using this ILP technique on a rich multisource language. The results show that our method greatly improves the feasibility and the efficiency of the process while staying accurate. They also confirm the benefits of using multiple sources to improve the diagnosis of cardiac arrhythmias.

Parallel ILP for distributed-memory architectures

The growth of machine-generated relational databases, both in the sciences and in industry, is rapidly outpacing our ability to extract useful information from them by manual means. This has brought into focus machine learning techniques like Inductive Logic Programming (ILP) that are able to extract human-comprehensible models for complex relational data. The price to pay is that ILP techniques are not efficient: they can be seen as performing a form of discrete optimisation, which is known to be computationally hard; and the complexity is usually some super-linear function of the number of examples. While little can be done to alter the theoretical bounds on the worst-case complexity of ILP systems, some practical gains may follow from the use of multiple processors. In this paper we survey the state-of-the-art on parallel ILP. We implement several parallel algorithms and study their performance using some standard benchmarks. The principal findings of interest are these: (1) of the techniques investigated, one that simply constructs models in parallel on each processor using a subset of data and then combines the models into a single one, yields the best results; and (2) sequential (approximate) ILP algorithms based on randomized searches have lower execution times than (exact) parallel algorithms, without sacrificing the quality of the solutions found.

Markov logic networks

We propose a simple approach to combining first-order logic and probabilistic graphical models in a single representation. A Markov logic network (MLN) is a first-order knowledge base with a weight attached to each formula (or clause). Together with a set of constants representing objects in the domain, it specifies a ground Markov network containing one feature for each possible grounding of a first-order formula in the KB, with the corresponding weight. Inference in MLNs is performed by MCMC over the minimal subset of the ground network required for answering the query. Weights are efficiently learned from relational databases by iteratively optimizing a pseudo-likelihood measure. Optionally, additional clauses are learned using inductive logic programming techniques. Experiments with a real-world database and knowledge base in a university domain illustrate the promise of this approach.

Induction from answer sets in nonmonotonic logic programs

Inductive logic programming (ILP) realizes inductive machine learning in computational logic. However, the present ILP mostly handles classical clausal programs, especially Horn logic programs, and has limited applications to learning nonmonotonic logic programs . This article studies a method for realizing induction in nonmonotonic logic programs. We consider an extended logic program as a background theory, and introduce techniques for inducing new rules using answer sets of the program. The produced new rules explain positive/negative examples in the context of inductive logic programming. The proposed methods extend the present ILP techniques to a syntactically and semantically richer framework, and contribute to a theory of nonmonotonic ILP.

Graph-based relational learning

Graph-based relational learning (GBRL) differs from logic-based relational learning, as addressed by inductive logic programming techniques, and differs from frequent subgraph discovery, as addressed by many graph-based data mining techniques. Learning from graphs, rather than logic, presents representational issues both in input data preparation and output pattern language. While a form of graph-based data mining, GBRL focuses on identifying novel, not necessarily most frequent, patterns in a graph-theoretic representation of data. This approach to graph-based data mining provides both simplifications and challenges over frequency-based approaches. In this paper we discuss these issues and future directions of graph-based relational learning.

New approach to pharmacophore mapping and QSAR analysis using inductive logic programming. Application to thermolysin inhibitors and glycogen phosphorylase B inhibitors.

A key problem in QSAR is the selection of appropriate descriptors to form accurate regression equations for the compounds under study. Inductive logic programming (ILP) algorithms are a class of machine-learning algorithms that have been successfully applied to a number of SAR problems. Unlike other QSAR methods, which use attributes to describe chemical structure, ILP uses relations. This gives ILP the advantages of not requiring explicit superimposition of individual compounds in a dataset, of dealing naturally with multiple conformations, and of using a language much closer to that used normally by chemists. We unify ILP and standard regression techniques to give a QSAR method that has the strength of ILP at describing steric structure with the familiarity and power of regression methods. Complex pharmacophores, correlating with activity, were identified and used as new indicator variables, along with the comparative molecular field analysis (CoMFA) prediction, to form predictive regression equations. We compared the formation of 3D-QSARs using standard CoMFA with the use of ILP on the well-studied thermolysin zinc protease inhibitor dataset and a glycogen phosphorylase inhibitor dataset. In each case the addition of ILP variables produced statistically better results (P < 0.01 for thermolysin and P < 0.05 for GP datasets) than the CoMFA analysis. Moreover, the new ILP variables were not found to increase the complexity of the final QSAR equations and gave possible insight into the binding mechanism of the ligand-protein complex under study.

Strategies in Combined Learning via Logic Programs

We discuss the adoption of a three-valued setting for inductive concept learning. Distinguishing between what is true, what is false and what is unknown can be useful in situations where decisions have to be taken on the basis of scarce, ambiguous, or downright contradictory information. In a three-valued setting, we learn a definition for both the target concept and its opposite, considering positive and negative examples as instances of two disjoint classes. To this purpose, we adopt Extended Logic Programs (ELP) under a Well-Founded Semantics with explicit negation (WFSX) as the representation formalism for learning, and show how ELPs can be used to specify combinations of strategies in a declarative way also coping with contradiction and exceptions. Explicit negation is used to represent the opposite concept, while default negation is used to ensure consistency and to handle exceptions to general rules. Exceptions are represented by examples covered by the definition for a concept that belong to the training set for the opposite concept. Standard Inductive Logic Programming techniques are employed to learn the concept and its opposite. Depending on the adopted technique, we can learn the most general or the least general definition. Thus, four epistemological varieties occur, resulting from the combination of most general and least general solutions for the positive and negative concept. We discuss the factors that should be taken into account when choosing and strategically combining the generality levels for positive and negative concepts. In the paper, we also handle the issue of strategic combination of possibly contradictory learnt definitions of a predicate and its explicit negation. All in all, we show that extended logic programs under well-founded semantics with explicit negation add expressivity to learning tasks, and allow the tackling of a number of representation and strategic issues in a principled way. Our techniques have been implemented and examples run on a state-of-the-art logic programming system with tabling which implements WFSX.

Inductive synthesis of recursive logic programs: achievements and prospects

The inductive synthesis of recursive logic programs from incomplete information, such as input/output examples, is a challenging subfield both of Inductive Logic Programming (ILP) and of the synthesis (in general) of logic programs, from formal specifications. We first overview past and present achievements, focusing on the techniques that were designed specifically for the inductive synthesis of recursive logic programs but also discussing a few general ILP techniques that can also induce non-recursive hypotheses. Then we analyse the prospects of these techniques in this task, investigating their applicability to software engineering as well as to knowledge acquisition and discovery.

AUTOMATICALLY EXPLORING HYPOTHESES ABOUT FAULT PREDICTION: A COMPARATIVE STUDY OF INDUCTIVE LOGIC PROGRAMMING METHODS

We evaluate a class of learning algorithms known as inductive logic programming (ILP) methods on the task of predicting fault density in C++ classes. Using these methods, a large space of possible hypotheses is searched in an automated fashion; further, the hypotheses are based directly on an abstract logical representation of the software, eliminating the need to manually propose numerical metrics that predict fault density. We compare two ILP systems, FOIL and FLIPPER, and conclude that FLIPPER generally outperforms FOIL on this problem. We analyze the reasons for the differing performance of these two systems, and based on the analysis, propose two extensions to FLIPPER: a user-directed bias towards easy-to-evaluate clauses, and an extension that allows FLIPPER to learn "counting clauses". Counting clauses augment logic programs with a variation of the "number restrictions" used in description logics, and significantly improve performance on this problem when prior knowledge is used. We also evaluate the use of ILP techniques for automatic generation of Boolean indicators and numeric metrics from the calling tree representation.

Inductive logic programming for relational knowledge discovery

Inductive logic programming (ILP) is concerned with the induction of logic programs from examples and background knowledge. In ILP, the shift of attention from program synthesis to knowledge discovery resulted in advanced techniques that are practically applicable for discovering knowledge in relational databases. This paper gives a brief introduction to ILP, presents selected ILP techniques for relational knowledge discovery and reviews selected ILP applications.

Scaling Up Inductive Logic Programming by Learning from Interpretations

When comparing inductive logic programming (ILP) and attribute-value learning techniques, there is a trade-off between expressive power and efficiency. Inductive logic programming techniques are typically more expressive but also less efficient. Therefore, the data sets handled by current inductive logic programming systems are small according to general standards within the data mining community. The main source of inefficiency lies in the assumption that several examples may be related to each other, so they cannot be handled independently. Within the learning from interpretations framework for inductive logic programming this assumption is unnecessary, which allows to scale up existing ILP algorithms. In this paper we explain this learning setting in the context of relational databases. We relate the setting to propositional data mining and to the classical ILP setting, and show that learning from interpretations corresponds to learning from multiple relations and thus extends the expressiveness of propositional learning, while maintaining its efficiency to a large extent (which is not the case in the classical ILP setting). As a case study, we present two alternative implementations of the ILP system TILDE (Top-down Induction of Logical DEcision trees): TILDEclassic, which loads all data in main memory, and TILDELDS, which loads the examples one by one. We experimentally compare the implementations, showing TILDELDS can handle large data sets (in the order of 100,000 examples or 100 MB) and indeed scales up linearly in the number of examples.