Grammatical Inference Research Articles

Improved genetic algorithm for the context-free grammatical inference

Inductive learning of formal languages, often called grammatical inference, is an active area inmachine learning and computational learning theory. By learning a language we understandfinding the grammar of the language when some positive (words from language) and negativeexamples (words that are not in language) are given. Learning mechanisms use the naturallanguage learning model: people master a language, used by their environment, by the analysis ofpositive and negative examples. The problem of inferring context-free languages (CFG) has boththeoretical and practical motivations. Practical applications include pattern recognition (forexample finding DTD or XML schemas for XML documents) and speech recognition (the abilityto infer context-free grammars for natural languages would enable speech recognition to modify itsinternal grammar on the fly). There were several attempts to find effective learning methods forcontext-free languages (for example [1,2,3,4,5]). In particular, Y.Sakakibara [3] introduced aninteresting method of finding a context-free grammar in the Chomsky normal form with a minimalset of nonterminals. He used the tabular representation similar to the parse table used in the CYKalgorithm, simultaneously with genetic algorithms. In this paper we present several adjustments tothe algorithm suggested by Sakakibara. The adjustments are concerned mainly with the geneticalgorithms used and are as follows:– we introduce a method of creating the initial population which makes use of characteristicfeatures of context-free grammars,– new genetic operations are used (mutation with a path added, ‘die process’, ‘war/diseaseprocess’),– different definition of the fitness function,– an effective compression of the structure of an individual in the population is suggested.These changes allow to speed up the process of grammar generation and, what is more, theyallow to infer richer grammars than considered in [3].

Swarm intelligence for natural language processing

Natural language processing NLP is an area dealing with computational methods for achieving human-like language processing. Traditionally, NLP research has been focused on developing efficient and robust algorithms to treat most NLP tasks, including syntactic and semantic analysis, grammar induction, summary and text generation, document clustering and machine translation. Swarm intelligence SI methods are effective to do so, since they have been successfully applied for many real-world problems. Recently, NLP and SI have been active areas of research, joined together more than once to solve problems in NLP field. This paper presents a review of recent developments of SI methods in NLP. It shows that only a few NLP tasks and applications were tackled by using SI-based algorithms. These mainly include text document clustering and classification, text summarisation, word sense disambiguation, information retrieval, and speaker recognition. This study also shows that four SI-based algorithms were examined in NLP field, including ant colony optimisation ACO, particle swarm optimisation PSO, bee swarm optimisation BSO, and firefly algorithm FA, emphasising ACO and PSO as the most investigated algorithms in this field.

Complexity of Grammar Induction for Quantum Types

Most categorical models of meaning use a functor from the syntactic category to the semantic category. When semantic information is available, the problem of grammar induction can therefore be defined as finding preimages of the semantic types under this forgetful functor, lifting the information flow from the semantic level to a valid reduction at the syntactic level. We study the complexity of grammar induction, and show that for a variety of type systems, including pivotal and compact closed categories, the grammar induction problem is NP-complete. Our approach could be extended to linguistic type systems such as autonomous or bi-closed categories.

Linguistic Models at the Crossroads of Agents, Learning and Formal Languages

This paper aims at reviewing the most relevant linguistic applications developed in the intersection between three different fields: machine learning, formal language theory and agent technologies. On the one hand, we present some of the main linguistic contributions of the intersection between machine learning and formal languages, which constitutes a well-established research area known as Grammatical Inference. On the other hand, we present an overview of the main linguistic applications of models developed in the intersection between agent technologies and formal languages, such as colonies, grammar systems and eco-grammar systems. Our goal is to show how interdisciplinary research between these three fields can contribute to better understand how natural language is acquired and processed.

Symbolic planning and control using game theory and grammatical inference

A system can accomplish an objective specified in temporal logic while interacting with an unknown, dynamic but rule-governed environment, by employing grammatical inference and adapting its plan of action on-line. The purposeful interaction of the system with its unknown environment can be described by a deterministic two-player zero-sum game. Using special new product operations, the whole game can be expressed with a factored, modular representation. This representation not only offers computational benefits but also isolates the unknown behavior of the dynamic environment in a particular subsystem, which then becomes the target of learning. As the fidelity of the identified environment model increases, the strategy synthesized based on the learned hypothesis converges in finite time to the one that satisfies the task specification.

Multilingual Dependency Parsing: Using Machine Translated Texts Instead of Parallel Corpora

Abstract This paper revisits the projection-based approach to dependency grammar induction task. Traditional cross-lingual dependency induction tasks one way or the other, depend on the existence of bitexts or target language tools such as part-of-speech (POS) taggers to obtain reasonable parsing accuracy. In this paper, we transfer dependency parsers using only approximate resources, i.e., machine translated bitexts instead of manually created bitexts. We do this by obtaining the the source side of the text from a machine translation (MT) system and then apply transfer approaches to induce parser for the target languages. We further reduce the need for the availability of labeled target language resources by using unsupervised target tagger. We show that our approach consistently outperforms unsupervised parsers by a bigger margin (8.2% absolute), and results in similar performance when compared with delexicalized transfer parsers.

A survey of grammatical inference in software engineering

Grammatical inference – used successfully in a variety of fields such as pattern recognition, computational biology and natural language processing – is the process of automatically inferring a grammar by examining the sentences of an unknown language. Software engineering can also benefit from grammatical inference. Unlike these other fields, which use grammars as a convenient tool to model naturally occurring patterns, software engineering treats grammars as first-class objects typically created and maintained for a specific purpose by human designers. We introduce the theory of grammatical inference and review the state of the art as it relates to software engineering.

Statistical syntax-based machine translation approach to diacritization problem

In this paper, the automatic diacritization of a language is modeled as a statistical syntax-based machine translation problem with the source undiacritized text and the target diacritized text of the same language. The grammatical inference technique ABL proposed in [2] is extended for learning a probabilistic synchronous context-free grammar from training corpus containing plain diacritized sentences only. The diacritization is to parse input sentences by the probabilistic CKY parsing algorithm for received grammar. This method is applied to Vietnamese with high quality result. As language independent building way, it can be applied to the other languages.

Learning Probabilistic Hierarchical Task Networks as Probabilistic Context-Free Grammars to Capture User Preferences

We introduce an algorithm to automatically learn probabilistic hierarchical task networks (pHTNs) that capture a user's preferences on plans by observing only the user's behavior. HTNs are a common choice of representation for a variety of purposes in planning, including work on learning in planning. Our contributions are twofold. First, in contrast with prior work, which employs HTNs to represent domain physics or search control knowledge, we use HTNs to model user preferences. Second, while most prior work on HTN learning requires additional information (e.g., annotated traces or tasks) to assist the learning process, our system only takes plan traces as input. Initially, we will assume that users carry out preferred plans more frequently, and thus the observed distribution of plans is an accurate representation of user preference. We then generalize to the situation where feasibility constraints frequently prevent the execution of preferred plans. Taking the prevalent perspective of viewing HTNs as grammars over primitive actions, we adapt an expectation-maximization (EM) technique from the discipline of probabilistic grammar induction to acquire probabilistic context-free grammars (pCFG) that capture the distribution on plans. To account for the difference between the distributions of possible and preferred plans, we subsequently modify this core EM technique by rescaling its input. We empirically demonstrate that the proposed approaches are able to learn HTNs representing user preferences better than the inside-outside algorithm. Furthermore, when feasibility constraints are obfuscated, the algorithm with rescaled input performs better than the algorithm with the original input.

NLP-inspired structural pattern recognition in chemical application

In this paper we report on a new structural pattern recognition approach for in silico prediction of chemical activity. It is based on grammatical inference on strings representing chemical compounds and string edit distance between a chemical compound and a formal grammar generalizing an activity class. In the late 1980s Weininger published a chemical language with a very simple and natural grammar. Recently, the algorithms suitable to process this language have been developed. From modeling of chemical activity with formal grammars and chemical compounds as words, a functionality is derivable to search for “structural alerts”, that is, molecular substructures and their combinatorial patterns that cause a molecule to have properties of interest. A biodegradability prediction system has been constructed to serve as an example throughout the paper. The source code and various files from the experiment are available from the corresponding author on request.

Adaptive Symbolic Control for Finite-State Transition Systems With Grammatical Inference

This note presents an approach that integrates elements from grammatical inference and game theory to address the problem of supervising finite-state transition systems operating in adversarial, partially known, rule-governed environments. The combined formulation produces controllers which guarantee that a transition system satisfies a task specification in the form of a logical formula, if and only 1) the true model of the environment is in the class of models inferable from positive data presentation (observations), 2) a characteristic sample of the environment's behavior is observed, and 3) the task specification is satisfiable given the capabilities described by the abstractions of the system and its environment.

Bayesian Constituent Context Model for Grammar Induction

Constituent Context Model (CCM) is an effective generative model for grammar induction, the aim of which is to induce hierarchical syntactic structure from natural text. The CCM simply defines the Multinomial distribution over constituents, which leads to a severe data sparse problem because long constituents are unlikely to appear in unseen data sets. This paper proposes a Bayesian method for constituent smoothing by defining two kinds of prior distributions over constituents: the Dirichlet prior and the Pitman-Yor Process prior. The Dirichlet prior functions as an additive smoothing method, and the PYP prior functions as a back-off smoothing method. Furthermore, a modified CCM is proposed to differentiate left constituents and right constituents in binary branching trees. Experiments show that both the proposed Bayesian smoothing method and the modified CCM are effective, and combining them attains or significantly improves the state-of-the-art performance of grammar induction evaluated on standard treebanks of various languages.

A Grammatical Inference Sequential Mining Algorithm for Protein Fold Recognition

Protein fold recognition plays an important role in computational protein analysis since it can determine protein function whose structure is unknown. In this paper, a Classified Sequential Pattern mining technique for Protein Fold Recognition (CSPF) is proposed. CSPF technique consists of two main phases: the sequential mining pattern phase and the fold recognition phase. In the sequential mining pattern phase, Mix & Test algorithm is developed based on Grammatical Inference, which is used as a training phase. Mix & Test algorithm minimizes I/O costs by one database scan, discovers subsequence combinations directly from sequences in memory without searching the whole sequence file, has no database projection, handles gaps, and works with variant length sequences without having to align them. In addition, a parallelized version of Mix & Test algorithm is applied to speed up Mix & Test algorithm performance. In the fold recognition phase, unknown protein folds are predicted via a proposed testing function. To test the performance, 36 SCOP protein folds are used, where the accuracy rate is 75.84% for training data and 59.7% for testing data.

Inference of Probabilistic Grammars in Different Rules Systems of Natural Languages

Abstract Grammar induction is an important and current area within computational linguistics. The results in induction of string transformation rules can be applied not only in linguistics but in many other pattern recognition tasks. The task of grammar induction belongs in general to NP -hard problems. The paper analyses the main rule types in natural languages and in string transformation systems. The paper focuses on the inference of inflectional rule systems which differs in many aspects from the traditional production rule system of Chomsky grammars. The work compares the main candidate methods on the learning of objective-case in the Hungarian language.

Introduction to the Special Issue on Grammatical Inference

As a field, Grammatical Inference addresses both theoretical and empirical learning problems, and the collection of papers within this special issue attests both to the diversity of these problems as well as the advances and insights that are being made by the researchers working within it. Thus we hope this special issue is of interest to the readership of Machine Learning.

An HDP Model for Inducing Combinatory Categorial Grammars

We introduce a novel nonparametric Bayesian model for the induction of Combinatory Categorial Grammars from POS-tagged text. It achieves state of the art performance on a number of languages, and induces linguistically plausible lexicons.

Parsing entire discourses as very long strings: Capturing topic continuity in grounded language learning

Grounded language learning, the task of mapping from natural language to a representation of meaning, has attracted more and more interest in recent years. In most work on this topic, however, utterances in a conversation are treated independently and discourse structure information is largely ignored. In the context of language acquisition, this independence assumption discards cues that are important to the learner, e.g., the fact that consecutive utterances are likely to share the same referent (Frank et al., 2013). The current paper describes an approach to the problem of simultaneously modeling grounded language at the sentence and discourse levels. We combine ideas from parsing and grammar induction to produce a parser that can handle long input strings with thousands of tokens, creating parse trees that represent full discourses. By casting grounded language learning as a grammatical inference task, we use our parser to extend the work of Johnson et al. (2012), investigating the importance of discourse continuity in children’s language acquisition and its interaction with social cues. Our model boosts performance in a language acquisition task and yields good discourse segmentations compared with human annotators.

Grammatical Inference for Modeling Mobility Patterns in Networks

Modeling of the mobility patterns arising in computer networks requires a compact and faithful representation of the mobility data collected from observations and measurements of the relevant network applications. This data can range from the information on the mobility of the agents that are being monitored by a wireless network to mobility information of nodes in mobile network applications. In this paper, we examine the use of probabilistic context-free grammars as the modeling framework for such data. We present a fast algorithm for deriving a concise probabilistic context-free grammar from the given training data. The algorithm uses an evaluation metric based on Bayesian formula for maximizing grammar a posteriori probability given the training data. We describe the application of this algorithm in two mobility modeling domains: 1) recognizing mobility patterns of monitored agents in different event data sets collected by sensor networks, and 2) modeling and generating node movements in mobile networks. We also discuss the model's performance in simulations utilizing both synthetic and real-world mobility traces.

Memetic cooperative coevolution of Elman recurrent neural networks

Cooperative coevolution decomposes an optimisation problem into subcomponents and collectively solves them using evolutionary algorithms. Memetic algorithms provides enhancement to evolutionary algorithms with local search. Recently, the incorporation of local search into a memetic cooperative coevolution method has shown to be efficient for training feedforward networks on pattern classification problems. This paper applies the memetic cooperative coevolution method for training recurrent neural networks on grammatical inference problems. The results show that the proposed method achieves better performance in terms of optimisation time and robustness.

Performance Evaluation of Methods for handling Premature Convergence in GA - Case of Grammar Induction

Genetic algorithms are meta-heuristic algorithms based on the biological evolution. These algorithms are found to be useful for finding near to optimum results for the NP-category of problems. GA suffers with the disadvantage of premature convergence. The paper focuses on the implementation of various techniques of handling premature convergence and the statistical evaluation of the obtained results to identify the optimal method to the problem of grammar induction. General Terms Evolutionary Computations, Grammar Inference, Learning.