Disjunctive Normal Form Formulas Research Articles

Rethinking Logic Minimization for Tabular Machine Learning

Tabular datasets can be viewed as logic functions that can be simplified using two-level logic minimization to produce minimal logic formulas in disjunctive normal form (DNF), which in turn can be readily viewed as an explainable decision rule set for binary classification. However, there are two problems with using logic minimization for tabular machine learning. First, tabular datasets often contain overlapping examples that have different class labels, which have to be resolved before logic minimization can be applied since logic minimization assumes <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">consistent</i> logic functions. Second, even without inconsistencies, logic minimization alone generally produces complex models with poor generalization because it exactly fits all data points, which leads to detrimental <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">overfitting</i> . How best to remove training instances to eliminate inconsistencies and overfitting is highly non-trivial. In this paper, we propose a novel statistical framework for removing these training samples so that logic minimization can become an effective approach to tabular machine learning. Using the proposed approach, we are able to obtain comparable performance as gradient boosted and ensemble decision trees, which have been the winning hypothesis classes in tabular learning competitions, but with human-understandable explanations in the form of decision rules. To our knowledge, neither logic minimization nor explainable decision rule methods have been able to achieve state-of-the-art performance before in tabular learning problems.

Differentiable learning of matricized DNFs and its application to Boolean networks

Boolean networks (BNs) are well-studied models of genomic regulation in biology where nodes are genes and their state transition is controlled by Boolean functions. We propose to learn Boolean functions as Boolean formulas in disjunctive normal form (DNFs) by an explainable neural network Mat_DNF and apply it to learning BNs. Directly expressing DNFs as a pair of binary matrices, we learn them using a single layer NN by minimizing a logically inspired non-negative cost function to zero. As a result, every parameter in the network has a clear meaning of representing a conjunction or literal in the learned DNF. Also we can prove that learning DNFs by the proposed approach is equivalent to inferring interpolants in logic between the positive and negative data. We applied our approach to learning three literature-curated BNs and confirmed its effectiveness. We also examine how generalization occurs when learning data is scarce. In doing so, we introduce two new operations that can improve accuracy, or equivalently generalizability for scarce data. The first one is to append a noise vector to the input learning vector. The second one is to continue learning even after learning error becomes zero. The first one is explainable by the second one. These two operations help us choose a learnable DNF, i.e., a root of the cost function, to achieve high generalizability.

Spatial state-action features for general games

In many board games and other abstract games, patterns have been used as features that can guide automated game-playing agents. Such patterns or features often represent particular configurations of pieces, empty positions, etc., which may be relevant for a game's strategies. Their use has been particularly prevalent in the game of Go, but also many other games used as benchmarks for AI research. In this paper, we formulate a design and efficient implementation of spatial state-action features for general games. These are patterns that can be trained to incentivise or disincentivise actions based on whether or not they match variables of the state in a local area around action variables. We provide extensive details on several design and implementation choices, with a primary focus on achieving a high degree of generality to support a wide variety of different games using different board geometries or other graphs. Secondly, we propose an efficient approach for evaluating active features for any given set of features. In this approach, we take inspiration from heuristics used in problems such as SAT to optimise the order in which parts of patterns are matched and prune unnecessary evaluations. This approach is defined for a highly general and abstract description of the problem—phrased as optimising the order in which propositions of formulas in disjunctive normal form are evaluated—and may therefore also be of interest to other types of problems than board games. An empirical evaluation on 33 distinct games in the Ludii general game system demonstrates the efficiency of this approach in comparison to a naive baseline, as well as a baseline based on prefix trees, and demonstrates that the additional efficiency significantly improves the playing strength of agents using the features to guide search.

MultiplexNet: Towards Fully Satisfied Logical Constraints in Neural Networks

We propose a novel way to incorporate expert knowledge into the training of deep neural networks. Many approaches encode domain constraints directly into the network architecture, requiring non-trivial or domain-specific engineering. In contrast, our approach, called MultiplexNet, represents domain knowledge as a quantifier-free logical formula in disjunctive normal form (DNF) which is easy to encode and to elicit from human experts. It introduces a latent Categorical variable that learns to choose which constraint term optimizes the error function of the network and it compiles the constraints directly into the output of existing learning algorithms. We demonstrate the efficacy of this approach empirically on several classical deep learning tasks, such as density estimation and classification in both supervised and unsupervised settings where prior knowledge about the domains was expressed as logical constraints. Our results show that the MultiplexNet approach learned to approximate unknown distributions well, often requiring fewer data samples than the alternative approaches. In some cases, MultiplexNet finds better solutions than the baselines; or solutions that could not be achieved with the alternative approaches. Our contribution is in encoding domain knowledge in a way that facilitates inference. We specifically focus on quantifier-free logical formulae that are specified over the output domain of a network. We show that this approach is both efficient and general; and critically, our approach guarantees 100% constraint satisfaction in a network's output.

Technical Perspective

Model counting is the problem of approximately counting the number |Sol(Φ)| of satisfying assignments to a given model Φ, which could, for example, be a formula in conjunctive normal form (CNF) or a formula in disjunctive normal form (DNF).

Approximation Algorithms for Stochastic Submodular Set Cover with Applications to Boolean Function Evaluation and Min-Knapsack

We present a new approximation algorithm for the stochastic submodular set cover (SSSC) problem called adaptive dual greedy . We use this algorithm to obtain a 3-approximation algorithm solving the stochastic Boolean function evaluation (SBFE) problem for linear threshold formulas (LTFs). We also obtain a 3-approximation algorithm for the closely related stochastic min-knapsack problem and a 2-approximation for a variant of that problem. We prove a new approximation bound for a previous algorithm for the SSSC problem, the adaptive greedy algorithm of Golovin and Krause. We also consider an approach to approximating SBFE problems using the adaptive greedy algorithm, which we call the Q -value approach. This approach easily yields a new result for evaluation of CDNF (conjunctive / disjunctive normal form) formulas, and we apply variants of it to simultaneous evaluation problems and a ranking problem. However, we show that the Q -value approach provably cannot be used to obtain a sublinear approximation factor for the SBFE problem for LTFs or read-once disjunctive normal form formulas.

Towards a HPC-oriented parallel implementation of a learning algorithm for bioinformatics applications.

BackgroundThe huge quantity of data produced in Biomedical research needs sophisticated algorithmic methodologies for its storage, analysis, and processing. High Performance Computing (HPC) appears as a magic bullet in this challenge. However, several hard to solve parallelization and load balancing problems arise in this context. Here we discuss the HPC-oriented implementation of a general purpose learning algorithm, originally conceived for DNA analysis and recently extended to treat uncertainty on data (U-BRAIN). The U-BRAIN algorithm is a learning algorithm that finds a Boolean formula in disjunctive normal form (DNF), of approximately minimum complexity, that is consistent with a set of data (instances) which may have missing bits. The conjunctive terms of the formula are computed in an iterative way by identifying, from the given data, a family of sets of conditions that must be satisfied by all the positive instances and violated by all the negative ones; such conditions allow the computation of a set of coefficients (relevances) for each attribute (literal), that form a probability distribution, allowing the selection of the term literals. The great versatility that characterizes it, makes U-BRAIN applicable in many of the fields in which there are data to be analyzed. However the memory and the execution time required by the running are of O(n3) and of O(n5) order, respectively, and so, the algorithm is unaffordable for huge data sets.ResultsWe find mathematical and programming solutions able to lead us towards the implementation of the algorithm U-BRAIN on parallel computers. First we give a Dynamic Programming model of the U-BRAIN algorithm, then we minimize the representation of the relevances. When the data are of great size we are forced to use the mass memory, and depending on where the data are actually stored, the access times can be quite different. According to the evaluation of algorithmic efficiency based on the Disk Model, in order to reduce the costs of the communications between different memories (RAM, Cache, Mass, Virtual) and to achieve efficient I/O performance, we design a mass storage structure able to access its data with a high degree of temporal and spatial locality. Then we develop a parallel implementation of the algorithm. We model it as a SPMD system together to a Message-Passing Programming Paradigm. Here, we adopt the high-level message-passing systems MPI (Message Passing Interface) in the version for the Java programming language, MPJ. The parallel processing is organized into four stages: partitioning, communication, agglomeration and mapping. The decomposition of the U-BRAIN algorithm determines the necessity of a communication protocol design among the processors involved. Efficient synchronization design is also discussed.ConclusionsIn the context of a collaboration between public and private institutions, the parallel model of U-BRAIN has been implemented and tested on the INTEL XEON E7xxx and E5xxx family of the CRESCO structure of Italian National Agency for New Technologies, Energy and Sustainable Economic Development (ENEA), developed in the framework of the European Grid Infrastructure (EGI), a series of efforts to provide access to high-throughput computing resources across Europe using grid computing techniques. The implementation is able to minimize both the memory space and the execution time. The test data used in this study are IPDATA (Irvine Primate splice- junction DATA set), a subset of HS3D (Homo Sapiens Splice Sites Dataset) and a subset of COSMIC (the Catalogue of Somatic Mutations in Cancer). The execution time and the speed-up on IPDATA reach the best values within about 90 processors. Then the parallelization advantage is balanced by the greater cost of non-local communications between the processors. A similar behaviour is evident on HS3D, but at a greater number of processors, so evidencing the direct relationship between data size and parallelization gain. This behaviour is confirmed on COSMIC. Overall, the results obtained show that the parallel version is up to 30 times faster than the serial one.

Some Investigations on Machine Learning Techniques for Automated Text Categorization

ABSTRACT The automated categorization (classification) of texts into predefined categories is one of the widely explored fields of research in text mining. Now-a-days, availability of digital data is very high, and to manage them in predefined categories has become a challenging task. Machine learning technique is an approach by which we can train automated classifier to classify the documents with minimum human assistance. This paper discusses the Naive Bayes, Rocchio, k-Nearest Neighborhood and Support Vector Machine methods within machine learning paradigm for automated text categorization of given documents in predefined categories. Keywords Machine learning, Text categorization. 1. INTRODUCTION In this cyber age, availability of digital document is increased drastically. Accessing the documents in convenient way has become difficult task as number and size of documents growing day by day. One such task is Text Categorization (TC), which means to label natural language text in predefined categories. Earlier Knowledge Engineering (KE) techniques were used for TC. KE is used in expert system which consists of manually defined logical rules of Disjunctive Normal Form (DNF) of type: if ( DNF formula) then (category); unlabeled documents into its appropriate category with high Document can be classified under particular category only if it satisfies the rule. The drawback of is this approach is the knowledge acquisition bottleneck. It is process in which expert person have to form DNF for new category. In last decade, the Machine Learning (ML) approach has gained popularity. In this approach, a general inductive process (learner) automatically builds a classifier. Learner automatically classifies document in predefined categories. Automated text categorization of documents in predefined categories is becoming popular in this digital age. Because now-a-days availability of digital documents increase dramatically, it becomes necessary to investigate and develop novel techniques for automated text categorization. Automated text categorization is applicable in document organization. In document organization documents has to be categorized in appropriate category. For example, in news paper agency the incoming advertisement has to be classified in one of the category like real estate, car for sale, office on rent. If such task is done manually than it would take lots of time. Automated systems are required that can accept the advertisement as input and categorize it to one of predefined categories. Let us take one more example where classifying the dynamic collection of text is to be done. Consider the example of e-mail filtering, where the computerized system is trained on “spam” mails to filter it out from non-spam mails [2]. Machine learning is an area of artificial intelligence. Machine learning deals with the study of methods for making computers learn like humans. Automated techniques from AI and machine learning have been developed to handle many problems of pattern recognition/categorization. One such task is text categorization of documents. Traditionally text categorization task is being carried out by KE techniques. In KE techniques human assistant is needed for forming decision rules for categorizing individual categories. So the idea is to explore the application of machine learning techniques for automated text categorization [2, 3], which can be free from human interactions. Automated text categorization with machine learning gained a prominent status in the information systems field. In this technique, a learner is implemented which automatically learn from previously classified documents. As discussed from this applications and importance of automated text categorization system encourages implementing computerized system which can classify incoming documents into appropriate predefined category. In implementing automated text categorization system the technique of machine learning algorithm which can be trained for some labeled documents and able to classify the incoming accuracy is to be explored. There are many machine learning algorithms available to build a learner for text categorization system. So it is interesting to implement few popular techniques of classification of text and to perform a comparative analysis in term of accuracy for such techniques. This paper deals with following objects. The first is to explore basic preprocessing steps for text categorization. It also presents the study of some machine learning techniques for text classification. The paper also focuses on investigation of performance issues in text categorization.

Contingent Planning as AND/OR Forward Search with Disjunctive Representation

This paper introduces a highly competitive contingent planner, that uses the novel idea of encoding belief states as disjunctive normal form formulae (To et al. 2009), for the search for solutions in the belief state space. In (To et al. 2009), a complete transition function for computing successor belief states in the presence of incomplete information has been defined. This work extends the function to handle non-deterministic and sensing actions in the AND/OR forward search paradigm for contingent planning solutions. The function allows one, under reasonable assumptions, to compute successor belief states efficiently, i.e., in polynomial time. The paper also presents a novel variant of an AND/OR search algorithm, called PrAO (Pruning AND/OR search), which allows the planner to significantly prune the search space; furthermore, by the time a solution is found, the remaining search graph is also the solution tree for the contingent planing problem. The strength of these techniques is confirmed by the empirical results obtained from a large set of benchmarks available in the literature.

A Conformant Planner with Explicit Disjunctive Representation of Belief States

This paper describes a novel and competitive complete conformant planner. Key to the enhanced performance is an efficient encoding of belief states as disjunctive normal form formulae and an efficient procedure for computing the successor belief state. We provide experimental comparative evaluation on a large pool of benchmarks. The novel design provides great efficiency and enhanced scalability, along with the intuitive structure of disjunctive normal form representations.

Polylogarithmic Independence Can Fool DNF Formulas

We show that any k-wise independent probability distribution on $\{0,1\}^n$ $O(m^{2.2}$ $2^{-\sqrt{k}/10})$-fools any boolean function computable by an m-clause disjunctive normal form (DNF) (or conjunctive normal form (CNF)) formula on n variables. Thus, for each constant $e>0$, there is a constant $c>0$ such that any boolean function computable by an m-clause DNF (or CNF) formula is $m^{-e}$-fooled by any $c\log^2m$-wise probability distribution. This resolves up to an $O(\log m)$ factor the depth-2 circuit case of a conjecture due to Linial and Nisan [Combinatorica, 10 (1990), pp. 349–365]. The result is equivalent to a new characterization of DNF (or CNF) formulas by low degree polynomials. It implies a similar statement for probability distributions with the small bias property. Using known explicit constructions of small probability spaces having the limited independence property or the small bias property, we directly obtain a large class of explicit pseudorandom generators of $O(\log^2m\log n)$-seed length for m-clause DNF (or CNF) formulas on n variables, improving previously known seed lengths.

Minimizing Disjunctive Normal Form Formulas and $AC^0$ Circuits Given a Truth Table

For circuit classes $R$, the fundamental computational problem Min-R asks for the minimum $R$-size of a Boolean function presented as a truth table. Prominent examples of this problem include Min-DNF, which asks whether a given Boolean function presented as a truth table has a $k$-term disjunctive normal form (DNF), and Min-Circuit (also called the minimum circuit size problem (MCSP)), which asks whether a Boolean function presented as a truth table has a size $k$ Boolean circuit. We present a new reduction proving that Min-DNF is NP-complete. It is significantly simpler than the known reduction of Masek [Some NP-Complete Set Covering Problems, manuscript, 1979], which is from Circuit-SAT. We then give a more complex reduction, yielding the result that Min-DNF cannot be approximated to within a factor smaller than $(\log N)^{\gamma}$, for some constant $\gamma>0$, assuming that NP is not contained in quasi-polynomial time. The standard greedy algorithm for Set Cover is often used in practice to approximate Min-DNF. The question of whether Min-DNF can be approximated to within a factor of $o(\log N)$ remains open, but we construct an instance of Min-DNF on which the solution produced by the greedy algorithm is $\Omega(\log N)$ larger than optimal. Finally, we turn to the question of approximating circuit size for slightly more general classes of circuits. DNF formulas are depth-two circuits of AND and OR gates. Depth-$d$ circuits are denoted by $AC^0_d$. We show that it is hard to approximate the size of $AC^0_d$ circuits (for large enough $d$) under cryptographic assumptions.

Clause/Term Resolution and Learning in the Evaluation of Quantified Boolean Formulas

Resolution is the rule of inference at the basis of most procedures for automated reasoning. In these procedures, the input formula is first translated into an equisatisfiable formula in conjunctive normal form (CNF) and then represented as a set of clauses. Deduction starts by inferring new clauses by resolution, and goes on until the empty clause is generated or satisfiability of the set of clauses is proven, e.g., because no new clauses can be generated. In this paper, we restrict our attention to the problem of evaluating Quantified Boolean Formulas (QBFs). In this setting, the above outlined deduction process is known to be sound and complete if given a formula in CNF and if a form of resolution, called ``Q-resolution'', is used. We introduce Q-resolution on terms, to be used for formulas in disjunctive normal form. We show that the computation performed by most of the available procedures for QBFs --based on the Davis-Logemann-Loveland procedure (DLL) for propositional satisfiability-- corresponds to a tree in which Q-resolution on terms and clauses alternate. This poses the theoretical bases for the introduction of learning, corresponding to recording Q-resolution formulas associated with the nodes of the tree. We discuss the problems related to the introduction of learning in DLL based procedures, and present solutions extending state-of-the-art proposals coming from the literature on propositional satisfiability. Finally, we show that our DLL based solver extended with learning, performs significantly better on benchmarks used in the 2003 QBF solvers comparative evaluation.

A Switching Lemma for Small Restrictions and Lower Bounds for k-DNF Resolution

We prove a new switching lemma that works for restrictions that set only a small fraction of the variables and is applicable to formulas in disjunctive normal form (DNFs) with small terms. We use this to prove lower bounds for the Res(k) propositional proof system, an extension of resolution which works with k-DNFs instead of clauses. We also obtain an exponential separation between depth d circuits of bottom fan-in k and depth d circuits of bottom fan-in k + 1. Our results for Res(k) are as follows: The 2n to n weak pigeonhole principle requires exponential size to refute in Res(k) for $k \leq \sqrt{\log n / \log \log n } $. For each constant k, there exists a constant w > k so that random w-CNFs require exponential size to refute in Res(k). For each constant k, there are sets of clauses which have polynomial size Res(k + 1) refutations but which require exponential size Res(k) refutations.

DNA-based algorithms for learning Boolean formulae

We apply a DNA-based massively parallel exhaustive search to solving the computational learning problems of DNF (disjunctive normal form) Boolean formulae. Learning DNF formulae from examples is one of the most important open problems in computational learning theory and the problem of learning 3-term DNF formulae is known as intractable if RP ≠ NP. We propose new methods to encode any k-term DNF formula to a DNA strand, evaluate the encoded DNF formula for a truth-value assignment by using hybridization and primer extension with DNA polymerase, and find a consistent DNF formula with the given examples. By employing these methods, we show that the class of k-term DNF formulae (for any constant k) and the class of general DNF formulae are efficiently learnable on DNA computer. Second, in order for the DNA-based learning algorithm to be robust for errors in the data, we implement the weighted majority algorithm on DNA computers, called DNA-based majority algorithm via amplification (DNAMA), which take a strategy of ``amplifying'' the consistent (correct) DNA strands. We show a theoretical analysis for the mistake bound of the DNA-based majority algorithm via amplification, and imply that the amplification to ``double the volumes'' of the correct DNA strands in the test tube works well.

Learning Monotone Log-Term DNF Formulas under the Uniform Distribution

Based on the uniform distribution PAC learning model, the learnability for the class of monotone disjunctive normal form formulas with at most O (log n ) terms, denoted O (log n )-term MDNF, is investigated. Using the technique of restriction, an algorithm that learns O (log n )-term MDNF by examples in polynomial time is given.

On Learning Read-k-Satisfy-j DNF

We study the learnability of read-k-satisfy-j (RkSj) DNF formulas. These are boolean formulas in disjunctive normal form (DNF), in which the maximum number of occurrences of a variable is bounded by k, and the number of terms satisfied by any assignment is at most j. After motivating the investigation of this class of DNF formulas, we present an algorithm that for any unknown RkSj DNF formula to be learned, with high probability finds a logically equivalent DNF formula using the well-studied protocol of equivalence and membership queries. The algorithm runs in polynomial time for $k\cdot j=O({\log n\over\log\log n})$, where n is the number of input variables.

Variable and term removal from Boolean formulae

Given a Boolean formula in disjunctive normal form, the variable deletion control set problem consists in finding a minimum cardinality set of variables whose deletion from the formula results in a DNF satisfying some prescribed property. Similar problems can be defined with respect to the fixation of variables or the deletion of terms in a DNF. In this paper, we investigate the complexity of such problems for a broad class of DNF properties.

Learning k-Term Monotone Boolean Formulae.

Valiant introduced a computational model of learning by examples, and gave the precise definition of polynominal time learnability based on the model. Since then, much effort has been devoted to characterize learnable classes of concepts on this model. Among such learnable classes is the one, denoted monotone k -term DNF, consisting of monotone disjunctive normal form formulae with at most k terms. So far it has been shown [6], [8] that for fixed k, monotone k -term DNF is learnable under the assumption that positive examples are drawn according to the uniform distribution. In this paper we introduce a class of probabilistic distributions, called smooth distributions, which is generalization of all the distribution classes which appeared in literature as the ones for specific distributiion setting: A smooth distribution is the one such that a ratio of the probabilities of any two examples with Hamming distance 1 is bounded from below by the inverse of some polynomial. It is proved that monotone k -term DNF is learnable even if positive examples are drawn according to smooth distributions. From this result it follows the learnability of monotone k -term DNF under specific distribution dealt with in literature such as product distributions [7] and q -bounded distributions [2].

On deterministic approximation of DNF

We develop several quasi-polynomial-time deterministic algorithms for approximating the fraction of truth assignments that satisfy a disjunctive normal form formula. The most efficient algorithm computes for a given DNF formulaF onn variables withm clauses and e > 0 an estimateY such that ¦Pr[F] −Y¦≤e in time which is $$(m\log (n))^{\exp (O(\sqrt {\log \log (m)} ))}$$ , for any constante. Although the algorithms themselves are deterministic, their analysis is probabilistic and uses the notion of limited independence between random variables.