Grammatical Inference Research Articles

Performance Evaluation of Methods for handling Premature Convergence in GA - Case of Grammar Induction

Genetic algorithms are meta-heuristic algorithms based on the biological evolution. These algorithms are found to be useful for finding near to optimum results for the NP-category of problems. GA suffers with the disadvantage of premature convergence. The paper focuses on the implementation of various techniques of handling premature convergence and the statistical evaluation of the obtained results to identify the optimal method to the problem of grammar induction. General Terms Evolutionary Computations, Grammar Inference, Learning.

Learning register automata: from languages to program structures

This paper reviews the development of Register Automaton learning, an enhancement of active automata learning to deal with infinite-state systems. We will revisit the precursor techniques and influences, which in total span over more than a decade. A large share of this development was guided and motivated by the increasingly popular application of grammatical inference techniques in the field of software engineering. We specifically focus on a key problem to achieve practicality in this field: the adequate treatment of data values ranging over infinite domains, a major source of undecidability. Starting with the first case studies, in which data was completely abstracted away, we revisit different steps towards dealing with data explicitly at a model level: we discuss Mealy machines as a model for systems with (data) output, automated alphabet abstraction refinement techniques as a two-dimensional extension of the partition-refinement based approach of active automata learning to also inferring optimal alphabet abstractions, and Register Mealy Machines, which can be regarded as programs restricted to data-independent data processing as it is typical for protocols or interface programs. We are convinced that this development will significantly contribute to paving the road for active automata learning to become a technology of high practical importance.

PAutomaC: a probabilistic automata and hidden Markov models learning competition

Approximating distributions over strings is a hard learning problem. Typical techniques involve using finite state machines as models and attempting to learn these; these machines can either be hand built and then have their weights estimated, or built by grammatical inference techniques: the structure and the weights are then learned simultaneously. The Probabilistic Automata learning Competition (PAutomaC), run in 2012, was the first grammatical inference challenge that allowed the comparison between these methods and algorithms. Its main goal was to provide an overview of the state-of-the-art techniques for this hard learning problem. Both artificial data and real data were presented and contestants were to try to estimate the probabilities of strings. The purpose of this paper is to describe some of the technical and intrinsic challenges such a competition has to face, to give a broad state of the art concerning both the problems dealing with learning grammars and finite state machines and the relevant literature. This paper also provides the results of the competition and a brief description and analysis of the different approaches the main participants used.

Guest editors introduction: special issue on Inductive Logic Programming (ILP 2012)

This issue contains selected papers from ILP 2012: the 22nd International Conference on Inductive Logic Programming held in Dubrovnik, Croatia on September 17–19, 2012. Out of the original 47 submissions to the conference, the authors of 8 selected peer-reviewed papers were invited to submit a revised and extended version for this special issue. As a result of an additional peer review adopting the journal criteria, 5 of these papers were finally accepted. The assorted papers demonstrate the vitality of current ILP research applied in challenging applications (here, object recognition), augmenting its logical foundations (metainterpretative learning, interpretation dynamics), and combined with extra-logical machinelearning techniques (neural networks, Bayes nets). In particular, the paper “Plane-based Object Categorization using Relational Learning” by R. Farid and C. Sammut, which won the best student-paper prize kindly sponsored by this journal, explores the hypothesis that relational description for classes of 3D objects can be built for robust object categorization in a real robotic application. Labelled sets of planes from 3D point clouds gathered during the RoboCup Rescue Robot competition are used as examples for an ILP system. The results show that ILP can be successfully applied to recognize objects encountered by a robot especially in an urban search and rescue environment. The article “Meta-interpretive Learning: Application to Grammatical Inference” by S.H. Muggleton, D. Lin, N. Pahlavi and A. Tamaddoni-Nezhad develops a framework in

Review of grammatical inference

Review of grammatical inference

Meta-interpretive learning: application to grammatical inference

Despite early interest Predicate Invention has lately been under-explored within ILP. We develop a framework in which predicate invention and recursive generalisations are implemented using abduction with respect to a meta-interpreter. The approach is based on a previously unexplored case of Inverse Entailment for Grammatical Inference of Regular languages. Every abduced grammar H is represented by a conjunction of existentially quantified atomic formulae. Thus ¬H is a universally quantified clause representing a denial. The hypothesis space of solutions for ¬H can be ordered by ?-subsumption. We show that the representation can be mapped to a fragment of Higher-Order Datalog in which atomic formulae in H are projections of first-order definite clause grammar rules and the existentially quantified variables are projections of first-order predicate symbols. This allows predicate invention to be effected by the introduction of first-order variables. Previous work by Inoue and Furukawa used abduction and meta-level reasoning to invent predicates representing propositions. By contrast, the present paper uses abduction with a meta-interpretive framework to invent relations. We describe the implementations of Meta-interpretive Learning (MIL) using two different declarative representations: Prolog and Answer Set Programming (ASP). We compare these implementations against a state-of-the-art ILP system MC-TopLog using the dataset of learning Regular and Context-Free grammars as well learning a simplified natural language grammar and a grammatical description of a staircase. Experiments indicate that on randomly chosen grammars, the two implementations have significantly higher accuracies than MC-TopLog. In terms of running time, Metagol is overall fastest in these tasks. Experiments indicate that the Prolog implementation is competitive with the ASP one due to its ability to encode a strong procedural bias. We demonstrate that MIL can be applied to learning natural grammars. In this case experiments indicate that increasing the available background knowledge, reduces the running time. Additionally ASPM (ASP using a meta-interpreter) is shown to have a speed advantage over Metagol when background knowledge is sparse. We also demonstrate that by combining MetagolR (Metagol with a Regular grammar meta-interpreter) and MetagolCF (Context-Free meta-interpreter) we can formulate a system, MetagolRCF, which can change representation by firstly assuming the target to be Regular, and then failing this, switch to assuming it to be Context-Free. MetagolRCF runs up to 100 times faster than MetagolCF on grammars chosen randomly from Regular and non-Regular Context-Free grammars.

User Behavior Analysis Using Alignment Based Grammatical Inference from Web Server Access Log

Application of data mining technique to the World Wide Web refers to as Web mining. Web based origination collects large volume of data for their operation. Analysis of such data can help the organization for better working (Marketing strategy, services, evaluation of effectiveness, promotional campaigns etc). This type of analysis require discovery of meaningful relationships from the large collection of primarily unstructured data stored in Web server access logs. We propose a new approach for automatically learning (context-free) grammar rules form server access log text (positive set) samples, based on the alignments between the sentences. Our approach works on pairs of unstructured sentences that have one or more words common.

Exploring the Rhythm of the Palestrina Style

This article presents a general-purpose formalism for modeling musical syntax as a probabilistic musical grammar. The formal probabilistic framework offers a precise yet flexible characterization of musical style as structure and process. Moreover, the grammar can be built algorithmically from a sample of musical examples, using a statistical grammar induction technique known as hidden Markov models (HMMs). The two fundamental assumptions of HMMs—termed finite memory and stationarity—are analyzed to show that the framework is expressive enough to capture a broad range of syntactic constraints in music. It is argued that the HMM technique draws its power from the ability to identify hidden structures that are important for shaping the musical surface. Thus, HMM-based grammar induction offers a practical, accurate, and methodologically sound tool for fine-grained modeling of musical style. As a concrete illustration, this article builds a formal grammar of rhythm for the Palestrina style. The grammar’s structure and components are carefully explained, and the formalism is compared with existing approaches to style characterization. Many traditional counterpoint rules are shown to naturally correspond with the grammar’s formal structure and are thus supported or refined. Other rules are disconfirmed or shown to lie outside the formalism’s scope. The long-standing problem of Renaissance meter is discussed in light of these results. Thus, through the Palestrina case study, the problem of grammar induction is framed in terms of traditional concerns in music scholarship in order to motivate application of the technique, illustrate its usefulness, and place it in a historical and methodological context within music theory research.

Modeling generalization and specialization with Extended Conceptual Graphs

Abstract The final goal of our research is to show that the performance of statistical rule induction can be improved by augmenting training data with semantic information. In order to prove this hypothesis, a statistical grammar induction system is to be created the knowledge base of which is represented by Extended Conceptual Graphs (ECGs). Since generalization and specialization are the basic operations of induction, they are of great significance in machine learning. As a consequence, the paper aims at investigating the least common generalization and the greatest common specialization of two ECG graphs. These operations should be traced back to the examination of ECG graph element instances. For this reason, a domain-specific ECG element instance type lattice (T′,≺) has been generated for the given test environment. Our final conclusion is that the least common generalization and the greatest common specialization of two ECG graphs always exist and can be computed. Therefore, the definition of the ≺ relation on element instances can be extended to a partial relation ⪯ on ECG diagram graphs, according to which F 1 ⪯ F 2 if graph Γ 1 is more specialized than Γ 2.

Improving Grammar Inference by a Memetic Algorithm

A memetic algorithm, a novel approach for solving NP-hard problems, has been applied in this paper for grammatical inference in the field of domain-specific languages (DSLs). DSLs are often designed by domain experts who have no knowledge about the syntax and semantics of programming languages. However, they are able to write sample programs to accomplish their goals and illustrate the features of their language. Grammatical inference is a technique to infer a context-free grammar from a set of positive (and negative) samples. This paper shows that grammatical inference may assist domain experts and software language engineers in developing DSLs by automatically producing a grammar, which describes a set of sample DSL programs. A memetic-algorithm-based tool is developed, which greatly improves results and robustness of the inference process.

Adaptive Bayesian HMM for Fully Unsupervised Chinese Part-of-Speech Induction

We propose an adaptive Bayesian hidden Markov model for fully unsupervised part-of-speech (POS) induction. The proposed model with its inference algorithm has two extensions to the first-order Bayesian HMM with Dirichlet priors. First our algorithm infers the optimal number of hidden states from the training corpus rather than fixes the dimensionality of state space beforehand. The second extension studies the Chinese unknown word processing module which measures similarities from both morphological properties and context distribution. Experimental results showed that both of these two extensions can help to find the optimal categories for Chinese in terms of both unsupervised clustering metrics and grammar induction accuracies on the Chinese Treebank.

Polynomial characteristic sets for DFA identification

We study the order in Grammatical Inference algorithms, and its influence on the polynomial (with respect to the data) identification of languages. This work is motivated by recent results on the polynomial convergence of data-driven grammatical inference algorithms. In this paper, we prove a sufficient condition that assures the existence of a characteristic sample whose size is polynomial with respect to the minimum DFA of the target language.

Hybrid system for handling premature convergence in GA – Case of grammar induction

Grammar induction (also known as grammar inference or language learning) is the process of learning grammar from training data. This paper presents hybrid genetic algorithm (GA) Tool for handling premature convergence for the case of grammar induction. The tool is developed in Java and the results are tested for the set of 20 languages. It is an effort towards introduction of a hybrid GA system where the collection of more than one type of reproduction methods is used for the grammar induction. Paper also includes the discussion on the various approaches used by author for handling the premature convergence problem in using GA.

Unsupervised dependency parsing without training

AbstractUsually unsupervised dependency parsers try to optimize the probability of a corpus by revising the dependency model that is assumed to have generated the corpus. In this paper we explore a different view in which a dependency structure is, among other things, a partial order on the nodes in terms of centrality or saliency. Under this assumption we directly model centrality and derive dependency trees from the ordering of words. The result is an approach to unsupervised dependency parsing that is very different from standard ones in that it requires no training data. The input words are ordered by centrality, and a parse is derived from the ranking using a simple deterministic parsing algorithm, relying on the universal dependency rules defined by Naseem et al. (Naseem, T., Chen, H., Barzilay, R., Johnson, M. 2010. Using universal linguistic knowledge to guide grammar induction. In Proceedings of Empirical Methods in Natural Language Processing, Boston, MA, USA, pp. 1234–44.). Our approach is evaluated on data from twelve different languages and is remarkably competitive.

Learning CFG using Improved TBL algorithm

Grammar Induction is the process of learning context free grammar from finite set of data of the positive and negative sample of language. The Problem of learning context free grammars from example has two aspects viz. determining the grammatical structure (topology) of the unknown grammar and identifying non-terminals. To solve these problems through a new hypothesis representation method called tabular representation is used , This method uses a table like data structure similar to the parse table used in the Cocke-Younger-Kasami parsing algorithm (CYK Algorithm) for context free grammar of Chomsky Normal Form. In this paper we introduce some improvement to the tabular representation algorithm (TBL) called Improved TBL algorithm (ITBL) dedicated to inference of grammar in Chomsky normal form. This ITBL algorithm uses a Genetic Algorithm (GA) to solve partitioning problem and focuses on the modified fitness function. It gives faster and optimal solutions for 5-8 string length.

Developing JSequitur to Study the Hierarchical Structure of Biological Sequences in a Grammatical Inference Framework of String Compression Algorithms

Grammatical inference methods are expected to find grammatical structures hidden in biological sequences. One hopes that studies of grammar serve as an appropriate tool for theory formation. Thus, we have developed JSequitur for automatically generating the grammatical structure of biological sequences in an inference framework of string compression algorithms. Our original motivation was to find any grammatical traits of several cancer genes that can be detected by string compression algorithms. Through this research, we could not find any meaningful unique traits of the cancer genes yet, but we could observe some interesting traits in regards to the relationship among gene length, similarity of sequences, the patterns of the generated grammar, and compression rate.

Information Extraction from the Un-Structured Document using Grammatical Inference and Alignment Similarity

Huge amount of information is available in un-structured (text) documents. Knowledge discovery in un-structured document has been recognized as promising task in the recent years. Since un-structured document is typically formatted for human viewing, it varies widely from document to document. Frequent changes made to their formatting further causes difficulty in construction of a global schema. So, Discovery of interesting rules form it is complex and tedious process. Most of the existing system uses hand-coded wrappers to extract information, which is monotonous and time consuming. In this paper we propose a novel and hybrid approach of learning (context-free) grammar rules that are based on alignment between texts. Also it automatically discovers the grammar rules using grammatical inference of repeated pattern present in un-structured (text) document. The generated rules can be used to infer the attribute value pairs from the unstructured text document.

Unsupervised grammar induction of clinical report sublanguage

BackgroundClinical reports are written using a subset of natural language while employing many domain-specific terms; such a language is also known as a sublanguage for a scientific or a technical domain. Different genres of clinical reports use different sublaguages, and in addition, different medical facilities use different medical language conventions. This makes supervised training of a parser for clinical sentences very difficult as it would require expensive annotation effort to adapt to every type of clinical text.MethodsIn this paper, we present an unsupervised method which automatically induces a grammar and a parser for the sublanguage of a given genre of clinical reports from a corpus with no annotations. In order to capture sentence structures specific to clinical domains, the grammar is induced in terms of semantic classes of clinical terms in addition to part-of-speech tags. Our method induces grammar by minimizing the combined encoding cost of the grammar and the corresponding sentence derivations. The probabilities for the productions of the induced grammar are then learned from the unannotated corpus using an instance of the expectation-maximization algorithm.ResultsOur experiments show that the induced grammar is able to parse novel sentences. Using a dataset of discharge summary sentences with no annotations, our method obtains 60.5% F-measure for parse-bracketing on sentences of maximum length 10. By varying a parameter, the method can induce a range of grammars, from very specific to very general, and obtains the best performance in between the two extremes.

Adapting modularity during learning in cooperative co-evolutionary recurrent neural networks

Adaptation during evolution has been an important focus of research in training neural networks. Cooperative coevolution has played a significant role in improving standard evolution of neural networks by organizing the training problem into modules and independently solving them. The number of modules required to represent a neural network is critical to the success of evolution. This paper proposes a framework for the adaptation of the number of modules during evolution. The framework is called adaptive modularity cooperative coevolution. It is used for training recurrent neural networks on grammatical inference problems. The results shows that the proposed approach performs better than its counterparts as the dimensionality of the problem increases.

EMBEDDING DSLS INTO GPLS: A GRAMMATICAL INFERENCE APPROACH *

Embedding of Domain-Specific Languages (DSLs) into General-Purpose Languages (GPLs) is oftenused to express domain-specific problems using the domain’s natural syntax inside GPL programs. It speeds up thedevelopment process, programs are more self-explanatory and repeating tasks are easier to handle. End-users ordomain experts know what the desired language syntax would look like, but do not know how to write a grammar andlanguage processing tools. Grammatical inference can be used for grammar extraction from input examples. Amemetic algorithm for grammatical inference, named MAGIc, was implemented to extract grammar from DSLexamples. In this work MAGIc is extended with embedding the inferred DSL into existing GPL grammar.Additionally, negative examples were also incorporated into the inference process. From the results it can be concludedthat MAGIc is successful for DSL embedding and that the inference process is improved with use of negativeexamples.http://dx.doi.org/10.5755/j01.itc.40.4.980