Natural Language Parsing Research Articles

Contentious Implementation and Retrenchment in Neoliberal Policy Reform: The Global Electric Power Industry, 1989–2001

We develop theory about the effect of domestic and global institutional forces across countries and over time, following a national government's adoption of a globally diffusing policy, on retrenchment—the degree to which a government reinstates the objectives of a policy's predecessor without repealing the new policy to balance conflicting institutional forces. World political culture legitimates and supports the new policy, while the policy's domestic opponents seek to mobilize opposition to it. Peer country governments' behavior and intergovernmental organizations may help or hinder domestic opponents' efforts. We tested our model by examining governments' renegotiation of the terms of private electricity generation projects in 62 countries in 1989–2001. Although no country formally repealed electricity liberalization during that period, governments selectively renegotiated the terms of private investment in roughly 20 percent of private power generation projects in countries that liberalized. Results support our hypotheses about the effects of domestic and global institutional forces—the former of which we measure through automated natural language parsing of 8.52 million newspaper articles—and the idea that domestic audiences' preexisting cognitive constructs and normative beliefs shape governments' implementation of globally diffusing policies.

A psycholinguistic model of natural language parsing implemented in simulated neurons

A natural language parser implemented entirely in simulated neurons is described. It produces a semantic representation based on frames. It parses solely using simulated fatiguing Leaky Integrate and Fire neurons, that are a relatively accurate biological model that is simulated efficiently. The model works on discrete cycles that simulate 10ms of biological time, so the parser has a simple mapping to psychological parsing time. Comparisons to human parsing studies show that the parser closely approximates this data. The parser makes use of Cell Assemblies and the semantics of lexical items is represented by overlapping hierarchical Cell Assemblies so that semantically related items share neurons. This semantic encoding is used to resolve prepositional phrase attachment ambiguities encountered during parsing. Consequently, the parser provides a neurally-based cognitive model of parsing.

10. Shortest path algorithms combined with natural language parsing, map pathways involved in neuroendocrine aging

10. Shortest path algorithms combined with natural language parsing, map pathways involved in neuroendocrine aging

Evaluating contributions of natural language parsers to protein–protein interaction extraction

Motivation: While text mining technologies for biomedical research have gained popularity as a way to take advantage of the explosive growth of information in text form in biomedical papers, selecting appropriate natural language processing (NLP) tools is still difficult for researchers who are not familiar with recent advances in NLP. This article provides a comparative evaluation of several state-of-the-art natural language parsers, focusing on the task of extracting protein–protein interaction (PPI) from biomedical papers. We measure how each parser, and its output representation, contributes to accuracy improvement when the parser is used as a component in a PPI system.Results: All the parsers attained improvements in accuracy of PPI extraction. The levels of accuracy obtained with these different parsers vary slightly, while differences in parsing speed are larger. The best accuracy in this work was obtained when we combined Miyao and Tsujii's Enju parser and Charniak and Johnson's reranking parser, and the accuracy is better than the state-of-the-art results on the same data.Availability: The PPI extraction system used in this work (AkanePPI) is available online at http://www-tsujii.is.s.u-tokyo.ac.jp/-100downloads/downloads.cgi. The evaluated parsers are also available online from each developer's site.Contact: yusuke@is.s.u-tokyo.ac.jp

A semiparametric generative model for efficient structured-output supervised learning

We present a semiparametric generative model for supervised learning with structured outputs. The main algorithmic idea is to replace the parameters of an underlying generative model (such as a stochastic grammars) with input-dependent predictions obtained by (kernel) logistic regression. This method avoids the computational burden associated with the comparison between target and predicted structure during the training phase, but requires as an additional input a vector of sufficient statistics for each training example. The resulting training algorithm is asymptotically more efficient than structured output SVM as the size of the output structure grows. At the same time, by computing parameters of a joint distribution as a function of the full input structure, typical expressiveness limitations of related conditional models (such as maximum entropy Markov models) can be potentially avoided. Empirical results on artificial and real data (in the domains of natural language parsing and RNA secondary structure prediction) show that the method works well in practice and scales up with the size of the output structures.

Polyadic Dynamic Logics for HPSG Parsing

Head-driven phrase structure grammar (HPSG) is one of the most prominent theories employed in deep parsing of natural language. Many linguistic theories are arguably best formalized in extensions of modal or dynamic logic (Keller, Feature logics, infinitary descriptions and grammar, 1993; Kracht, Linguistics Philos 18:401---458, 1995; Moss and Tiede, In: Blackburn, van Benthem, and Wolther (eds.) Handbook of modal logic, 2006), and HPSG seems to be no exception. Adequate extensions of dynamic logic have not been studied in detail, however; the most important aspect is the reference to sets of substructures. In this paper, an adequate extension is identified, and some important results are established: Satisfiability is highly undecidable, and model checking is shown to be in EXPTIME and PSPACE-hard. A fragment with polynomial time model checking procedures is identified; it is shown to cover considerable fragments of HPSG.

Algorithms for Deterministic Incremental Dependency Parsing

Parsing algorithms that process the input from left to right and construct a single derivation have often been considered inadequate for natural language parsing because of the massive ambiguity typically found in natural language grammars. Nevertheless, it has been shown that such algorithms, combined with treebank-induced classifiers, can be used to build highly accurate disambiguating parsers, in particular for dependency-based syntactic representations. In this article, we first present a general framework for describing and analyzing algorithms for deterministic incremental dependency parsing, formalized as transition systems. We then describe and analyze two families of such algorithms: stack-based and list-based algorithms. In the former family, which is restricted to projective dependency structures, we describe an arc-eager and an arc-standard variant; in the latter family, we present a projective and a non-projective variant. For each of the four algorithms, we give proofs of correctness and complexity. In addition, we perform an experimental evaluation of all algorithms in combination with SVM classifiers for predicting the next parsing action, using data from thirteen languages. We show that all four algorithms give competitive accuracy, although the non-projective list-based algorithm generally outperforms the projective algorithms for languages with a non-negligible proportion of non-projective constructions. However, the projective algorithms often produce comparable results when combined with the technique known as pseudo-projective parsing. The linear time complexity of the stack-based algorithms gives them an advantage with respect to efficiency both in learning and in parsing, but the projective list-based algorithm turns out to be equally efficient in practice. Moreover, when the projective algorithms are used to implement pseudo-projective parsing, they sometimes become less efficient in parsing (but not in learning) than the non-projective list-based algorithm. Although most of the algorithms have been partially described in the literature before, this is the first comprehensive analysis and evaluation of the algorithms within a unified framework.

Probabilistic Context-Free Grammars Estimated from Infinite Distributions

In this paper, we consider probabilistic context-free grammars, a class of generative devices that has been successfully exploited in several applications of syntactic pattern matching, especially in statistical natural language parsing. We investigate the problem of training probabilistic context-free grammars on the basis of distributions defined over an infinite set of trees or an infinite set of sentences by minimizing the cross-entropy. This problem has applications in cases of context-free approximation of distributions generated by more expressive statistical models. We show several interesting theoretical properties of probabilistic context-free grammars that are estimated in this way, including the previously unknown equivalence between the grammar cross-entropy with the input distribution and the so-called derivational entropy of the grammar itself. We discuss important consequences of these results involving the standard application of the maximum-likelihood estimator on finite tree and sentence samples, as well as other finite-state models such as Hidden Markov Models and probabilistic finite automata.

High-Performance Training of Conditional Random Fields for Large-Scale Applications of Labeling Sequence Data

Conditional random fields (CRFs) have been successfully applied to various applications of predicting and labeling structured data, such as natural language tagging & parsing, image segmentation & object recognition, and protein secondary structure prediction. The key advantages of CRFs are the ability to encode a variety of overlapping, non-independent features from empirical data as well as the capability of reaching the global normalization and optimization. However, estimating parameters for CRFs is very time-consuming due to an intensive forward-backward computation needed to estimate the likelihood function and its gradient during training. This paper presents a high-performance training of CRFs on massively parallel processing systems that allows us to handle huge datasets with hundreds of thousand data sequences and millions of features. We performed the experiments on an important natural language processing task (text chunking) on large-scale corpora and achieved significant results in terms of both the reduction of computational time and the improvement of prediction accuracy.

Improved Identification of Noun Phrases in Clinical Radiology Reports Using a High-Performance Statistical Natural Language Parser Augmented with the UMLS Specialist Lexicon

ObjectiveThe aim of this study was to develop and evaluate a method of extracting noun phrases with full phrase structures from a set of clinical radiology reports using natural language processing (NLP) and to investigate the effects of using the UMLS® Specialist Lexicon to improve noun phrase identification within clinical radiology documents. DesignThe noun phrase identification (NPI) module is composed of a sentence boundary detector, a statistical natural language parser trained on a nonmedical domain, and a noun phrase (NP) tagger. The NPI module processed a set of 100 XML-represented clinical radiology reports in Health Level 7 (HL7)® Clinical Document Architecture (CDA)–compatible format. Computed output was compared with manual markups made by four physicians and one author for maximal (longest) NP and those made by one author for base (simple) NP, respectively. An extended lexicon of biomedical terms was created from the UMLS Specialist Lexicon and used to improve NPI performance. ResultsThe test set was 50 randomly selected reports. The sentence boundary detector achieved 99.0% precision and 98.6% recall. The overall maximal NPI precision and recall were 78.9% and 81.5% before using the UMLS Specialist Lexicon and 82.1% and 84.6% after. The overall base NPI precision and recall were 88.2% and 86.8% before using the UMLS Specialist Lexicon and 93.1% and 92.6% after, reducing false-positives by 31.1% and false-negatives by 34.3%. ConclusionThe sentence boundary detector performs excellently. After the adaptation using the UMLS Specialist Lexicon, the statistical parser's NPI performance on radiology reports increased to levels comparable to the parser's native performance in its newswire training domain and to that reported by other researchers in the general nonmedical domain.

Lazy functional programming in Java

In this paper, we show how lazy functional programming techniques can be used within the Java programming language. We provide Java implementations of classic examples of lazy lists, such as the Sieve of Eratosthenes, the Eight Queens Problem, and natural-language parsing. We discuss how well these implementations succeed, compared to their original counterparts. We also point out the potential synergy between adding lazy techniques to Java, and adding generic types. The examples we provide would be suitable for teaching functional programming concepts in the context of a Java-based syllabus.

Ambiguity Resolution Analysis in Incremental Parsing of Natural Language

Incremental parsing gains its importance in natural language processing and psycholinguistics because of its cognitive plausibility. Modeling the associated cognitive data structures, and their dynamics, can lead to a better understanding of the human parser. In earlier work, we have introduced a recursive neural network (RNN) capable of performing syntactic ambiguity resolution in incremental parsing. In this paper, we report a systematic analysis of the behavior of the network that allows us to gain important insights about the kind of information that is exploited to resolve different forms of ambiguity. In attachment ambiguities, in which a new phrase can be attached at more than one point in the syntactic left context, we found that learning from examples allows us to predict the location of the attachment point with high accuracy, while the discrimination amongst alternative syntactic structures with the same attachment point is slightly better than making a decision purely based on frequencies. We also introduce several new ideas to enhance the architectural design, obtaining significant improvements of prediction accuracy, up to 25% error reduction on the same dataset used in previous work. Finally, we report large scale experiments on the entire Wall Street Journal section of the Penn Treebank. The best prediction accuracy of the model on this large dataset is 87.6%, a relative error reduction larger than 50% compared to previous results.

Reasoning about inconsistencies in natural language requirements

The use of logic in identifying and analyzing inconsistency in requirements from multiple stakeholders has been found to be effective in a number of studies. Nonmonotonic logic is a theoretically well-founded formalism that is especially suited for supporting the evolution of requirements. However, direct use of logic for expressing requirements and discussing them with stakeholders poses serious usability problems, since in most cases stakeholders cannot be expected to be fluent with formal logic. In this article, we explore the integration of natural language parsing techniques with default reasoning to overcome these difficulties. We also propose a method for automatically discovering inconsistencies in the requirements from multiple stakeholders, using both theorem-proving and model-checking techniques, and show how to deal with them in a formal manner. These techniques were implemented and tested in a prototype tool calledCARL. The effectiveness of the techniques and of the tool are illustrated by a classic example involving conflicting requirements from multiple stakeholders.

Discriminative Reranking for Natural Language Parsing

This article considers approaches which rerank the output of an existing probabilistic parser. The base parser produces a set of candidate parses for each input sentence, with associated probabilities that define an initial ranking of these parses. A second model then attempts to improve upon this initial ranking, using additional features of the tree as evidence. The strength of our approach is that it allows a tree to be represented as an arbitrary set of features, without concerns about how these features interact or overlap and without the need to define a derivation or a generative model which takes these features into account. We introduce a new method for the reranking task, based on the boosting approach to ranking problems described in Freund et al. (1998). We apply the boosting method to parsing the Wall Street Journal treebank. The method combined the log-likelihood under a baseline model (that of Collins [1999]) with evidence from an additional 500,000 features over parse trees that were not included in the original model. The new model achieved 89.75% F-measure, a 13% relative decrease in F-measure error over the baseline model's score of 88.2%. The article also introduces a new algorithm for the boosting approach which takes advantage of the sparsity of the feature space in the parsing data. Experiments show significant efficiency gains for the new algorithm over the obvious implementation of the boosting approach. We argue that the method is an appealing alternative-in terms of both simplicity and efficiency-to work on feature selection methods within log-linear (maximum-entropy) models. Although the experiments in this article are on natural language parsing (NLP), the approach should be applicable to many other NLP problems which are naturally framed as ranking tasks, for example, speech recognition, machine translation, or natural language generation.

The influence of contextual contrast on syntactic processing: evidence for strong-interaction in sentence comprehension

The present study compares the processing of unambiguous restrictive and non-restrictive relative clauses (RCs) within both a null context and a supportive discourse using a self-paced reading methodology. Individuals read restrictive RCs more slowly than non-restrictive RCs in a null context, but processed restrictive RCs faster than non-restrictive RCs in supportive context, resulting in an interaction between context and RC type. These results provide evidence for two theoretical points. First, principles analogous to those in referential theory [Altmann G. T. M., & Steedman, M. (1988). Interaction with context during human sentence processing. Cognition, 30, 191–238; Crain, S., & Steedman, M. (1985). On not being led up the garden path: The use of context by the psychological parser. In D. Dowty, L. Karttunnen, A. Zwicky (Eds.), Natural language parsing. Cambridge, UK: Cambridge University Press] apply not only in resolving ambiguity but also in processing unambiguous sentences. Second, the discourse context can guide and facilitate interpretive processing. This result suggests that intrasentential factors such as syntax are not autonomous from contextual processing, contrary to the modularity hypothesis [Fodor, J. A. (1983). Modularity of mind. Cambridge, MA: MIT Press].

Managing educational resource in medicine: system design and integration

The objective of this paper is to describe our experience in developing a tool based on web technologies, for storing, managing and providing medical courses written by professors in the medical university of Rennes. The increasing number of documents sent by professors leaded us to build a specific resource management system. We created a relational database, containing all meta information about each available document. Professors provide their courses in various formats. We use natural language parsing techniques to extract information from the text, and provide a proper semantic indexation which will be used by a medical-specific search engine. Then the content of our database is dynamically displayed on a web interface. A user's directory identifies teachers and students, controls the access, tracks the students' navigation and allows an on-line discussion forum. This portal contains 524 courses and we had more than 3,000,000 connections on it last year. We are now integrating its content using the semantic web approach in a larger project: the French Virtual Medical University.

The role of discourse context in the processing of a flexible word-order language

On-line comprehension studies of flexible word-order languages find that noncanonical (‘scrambled’) structures induce more difficulty than canonical structures [e.g., Hyönä & Hujanen, Q. J. Exp. Psychol. 50A (1997) 841–858], with this difference being attributed to the structural complexity/infrequency of these forms. However, by presenting sentences in isolation, most existing studies have not examined the extent to which processing patterns can be explained by discourse factors. In Finnish, an articleless flexible word-order language with canonical SVO order, word order can be used to encode the given/new distinctions: OVS marks the object as given and the subject as new; SVO is more flexible, being used in multiple contexts. Thus, the observed dispreference for noncanonical orders in previous work may arise from additional presuppositions needed to understand them out of context [cf. Crain & Steedman, Natural Language Parsing, (1985), 320–358]. Two experiments were conducted to examine these issues in Finnish. Experiment 1 used a self-paced reading task to show that the usual difficulty associated with noncanonical constructions is partially alleviated in the presence of appropriate discourse contexts. In Experiment 2, the eye gaze of listeners was tracked as they heard spoken descriptions of scenes, so as to test whether specific on-line referential processes are involved in SVO/OVS comprehension: Upon hearing an OV… sequence, Finnish listeners should expect the upcoming noun to be discourse-new, whereas an SV… sequence makes no such prediction. The results confirmed these predictions. As compared to SVO, OVS sentences showed anticipatory eye movements to a discourse-new referent at the second noun onset, even before participants had enough acoustic information to recognize this word. Our findings illustrate that in a flexible word-order language, a noncanonical order can result in anticipatory processes regarding the discourse status of a yet-to-be-heard constituent.

Estimating probabilities for unbounded categorization problems

We propose two output activation functions for estimating probability distributions over an unbounded number of categories with a recurrent neural network, and derive the statistical assumptions which they embody. Both these methods perform better than the standard approach to such problems, when applied to probabilistic parsing of natural language with Simple Synchrony Networks.

Automatic control of household activity using speech recognition and natural language

Speech recognition and natural language parsing components are used to extract the meaning of the user's spoken input. The system stores a semantic representation of an electronic activity guide, and the contents of the guide can be mapped into the grammars used by the natural language parser. Thus, when the user wishes to navigate through the complex menu structure of the electronic activity guide, he or she only needs to speak in natural language sentences. The system automatically filters the contents of the guide and supplies the user with on-screen display or synthesized speech responses to the user's request. The system allows the user to communicate in a natural way with a variety of devices communicating with the home network or home gateway.

The On-Line Resolution of the Sentence Complement/Relative Clause Ambiguity: Evidence from Spanish

The On-Line Resolution of the Sentence Complement/Relative Clause Ambiguity: Evidence from Spanish