Maximum Entropy Classifier Research Articles

A Topic-Based Coherence Model for Statistical Machine Translation

Coherence that ties sentences of a text into a meaningfully connected structure is of great importance to text generation and translation. In this paper, we propose a topic-based coherence model to produce coherence for document translation, in terms of the continuity of sentence topics in a text. We automatically extract a coherence chain for each source text to be translated. Based on the extracted source coherence chain, we adopt a maximum entropy classifier to predict the target coherence chain that defines a linear topic structure for the target document. The proposed topic-based coherence model then uses the predicted target coherence chain to help decoder select coherent word/phrase translations. Our experiments show that incorporating the topic-based coherence model into machine translation achieves substantial improvement over both the baseline and previous methods that integrate document topics rather than coherence chains into machine translation.

Multitechnique paraphrase alignment

This work uses parallel monolingual corpora for a detailed study of the task of sub-sentential paraphrase acquisition. We argue that the scarcity of this type of resource is compensated by the fact that it is the most suited type for studies on paraphrasing. We propose a large exploration of this task with experiments on two languages with five different acquisition techniques, selected for their complementarity, their combinations, as well as four monolingual corpus types of varying comparability. We report, under all conditions, a significant improvement over all techniques by validating candidate paraphrases using a maximum entropy classifier. An important result of our study is the identification of difficult-to-acquire paraphrase pairs, which are classified and quantified in a bilingual typology.

Integrative Semantic Dependency Parsing via Efficient Large-scale Feature Selection

Semantic parsing, i.e., the automatic derivation of meaning representation such as an instantiated predicate-argument structure for a sentence, plays a critical role in deep processing of natural language. Unlike all other top systems of semantic dependency parsing that have to rely on a pipeline framework to chain up a series of submodels each specialized for a specific subtask, the one presented in this article integrates everything into one model, in hopes of achieving desirable integrity and practicality for real applications while maintaining a competitive performance. This integrative approach tackles semantic parsing as a word pair classification problem using a maximum entropy classifier. We leverage adaptive pruning of argument candidates and large-scale feature selection engineering to allow the largest feature space ever in use so far in this field, it achieves a state-of-the-art performance on the evaluation data set for CoNLL-2008 shared task, on top of all but one top pipeline system, confirming its feasibility and effectiveness.

Automatic Detection of Preposition Errors in Learner Writing

In this article, we present an approach to the automatic correction of preposition errors in L2 English. Our system, based on a maximum entropy classifier, achieves average precision of 42% and recall of 35% on this task. The discussion of results obtained on correct and incorrect data aims to establish what characteristics of L2 writing prove particularly problematic in this task.

The ontogene system: an advanced information extraction application for biological literature

The ontogene system: an advanced information extraction application for biological literature

Feature selection for sentiment analysis based on content and syntax models

Recent solutions for sentiment analysis have relied on feature selection methods ranging from lexicon-based approaches where the set of features are generated by humans, to approaches that use general statistical measures where features are selected solely on empirical evidence. The advantage of statistical approaches is that they are fully automatic, however, they often fail to separate features that carry sentiment from those that do not. In this paper we propose a set of new feature selection schemes that use a Content and Syntax model to automatically learn a set of features in a review document by separating the entities that are being reviewed from the subjective expressions that describe those entities in terms of polarities. By focusing only on the subjective expressions and ignoring the entities, we can choose more salient features for document-level sentiment analysis. The results obtained from using these features in a maximum entropy classifier are competitive with the state-of-the-art machine learning approaches.

Development of a benchmark corpus to support the automatic extraction of drug-related adverse effects from medical case reports

A significant amount of information about drug-related safety issues such as adverse effects are published in medical case reports that can only be explored by human readers due to their unstructured nature. The work presented here aims at generating a systematically annotated corpus that can support the development and validation of methods for the automatic extraction of drug-related adverse effects from medical case reports. The documents are systematically double annotated in various rounds to ensure consistent annotations. The annotated documents are finally harmonized to generate representative consensus annotations. In order to demonstrate an example use case scenario, the corpus was employed to train and validate models for the classification of informative against the non-informative sentences. A Maximum Entropy classifier trained with simple features and evaluated by 10-fold cross-validation resulted in the F1 score of 0.70 indicating a potential useful application of the corpus.

Size distribution and settling velocities of suspended particles in a tidal embayment

Field studies were carried out to investigate seasonal and bay-wide variations in the particle size distributions (PSDs) and settling velocities of suspended particles in Newport Bay, the second largest tidal embayment in southern California. Maximum Entropy Classification (MEC) of the data identified three PSD groups: (1) suspended particles with a single mode around 10–20 μm (Group 1), (2) suspended particles with a single mode similar to Group 1, but shifted to smaller particle sizes (Group 2), and (3) suspended particles with a coarse mode at around 100 μm (Group 3). The three PSD groups have distinct seasonal and spatial patterns, and different size-settling velocity relationships, consistent with the hypothesis that Newport Bay longitudinally fractionates allochthonous particles from its tributaries by both size and settling velocity, and generates large and fast settling autochthonous particles, probably in the form of biological debris. Particle concentrations in Groups 1 and 2 are significantly correlated with fecal indicator bacteria, suggesting possible linkages between fecal pollution and particle transfer in this tidal embayment.

Text Categorization Based on Emergency Domain Words: A System Engineering View

Most of the research on text categorization didnot consider the characteristics of the emergency domain. Considering the characters of a specific emergency domain, we propose a text classification based on emergency domain words and machine learning technique taking a System Engineering view. With CHI as evaluation function to select text features, the addition of emergency domain words, Maximum Entropy classifier and KNN classifier, we conduct a series of experiments on emergency event texts classification. The experiments show that, the introduction of emergency domain words will increase the average accuracy of maximum entropy classifier and KNN classifier by 4% to 5%. Particularly maximum entropy classifier can still get an average accuracy rate as 97.0% after the introduction of the emergency domain terms.

Broadcast News Story Segmentation Using Conditional Random Fields and Multimodal Features

In this paper, we propose integration of multimodal features using conditional random fields (CRFs) for the segmentation of broadcast news stories. We study story boundary cues from lexical, audio and video modalities, where lexical features consist of lexical similarity, chain strength and overall cohesiveness; acoustic features involve pause duration, pitch, speaker change and audio event type; and visual features contain shot boundaries, anchor faces and news title captions. These features are extracted in a sequence of boundary candidate positions in the broadcast news. A linear-chain CRF is used to detect each candidate as boundary/non-boundary tags based on the multimodal features. Important interlabel relations and contextual feature information are effectively captured by the sequential learning framework of CRFs. Story segmentation experiments show that the CRF approach outperforms other popular classifiers, including decision trees (DTs), Bayesian networks (BNs), naive Bayesian classifiers (NBs), multilayer perception (MLP), support vector machines (SVMs) and maximum entropy (ME) classifiers. key words: story segmentation, conditional random fields

Emotion Detection in Suicide Notes using Maximum Entropy Classification

An ensemble of supervised maximum entropy classifiers can accurately detect and identify sentiments expressed in suicide notes. Using lexical and syntactic features extracted from a training set of externally annotated suicide notes, we trained separate classifiers for each of fifteen pre-specified emotions. This formed part of the 2011 i2b2 NLP Shared Task, Track 2. The precision and recall of these classifiers related strongly with the number of occurrences of each emotion in the training data. Evaluating on previously unseen test data, our best system achieved an F1 score of 0.534.

Maxent: An R Package for Low-memory Multinomial Logistic Regression with Support for Semi-automated Text Classification

maxent is a package with tools for data classification using multinomial logistic regression, also known as maximum entropy. The focus of this maximum entropy classifier is to minimize memory consumption on very large datasets, particularly sparse document-term matrices represented by the tm text mining package.

Utilizing Various Natural Language Processing Techniques for Biomedical Interaction Extraction

The vast number of biomedical literature is an important source of biomedical interaction information discovery. However, it is complicated to obtain interaction information from them because most of them are not easily readable by machine. In this paper, we present a method for extracting biomedical interaction information assuming that the biomedical Named Entities (NEs) are already identified. The proposed method labels all possible pairs of given biomedical NEs as INTERACTION or NO-INTERACTION by using a Maximum Entropy (ME) classifier. The features used for the classifier are obtained by applying various NLP techniques such as POS tagging, base phrase recognition, parsing and predicate-argument recognition. Especially, specific verb predicates (activate, inhibit, diminish and etc.) and their biomedical NE arguments are very useful features for identifying interactive NE pairs. Based on this, we devised a twostep method: 1) an interaction verb extraction step to find biomedically salient verbs, and 2) an argument relation identification step to generate partial predicate-argument structures between extracted interaction verbs and their NE arguments. In the experiments, we analyzed how much each applied NLP technique improves the performance. The proposed method can be completely improved by more than 2% compared to the baseline method. The use of external contextual features, which are obtained from outside of NEs, is crucial for the performance improvement. We also compare the performance of the proposed method against the co-occurrence-based and the rule-based methods. The result demonstrates that the proposed method considerably improves the performance.

Hybrid learning of Syntactic and Semantic Dependencies

This paper presents our solution for jointly parsing of syntactic and semantic dependencies. The Maximum Entropy (ME) classifier is selected in this system. Also the Mutual Information (MI) model was utilized into feature selection of dependency labeling. Results show that the MI model allows the system to get better performance and reduce training hours.

Feature selection techniques for maximum entropy based biomedical named entity recognition

Named entity recognition is an extremely important and fundamental task of biomedical text mining. Biomedical named entities include mentions of proteins, genes, DNA, RNA, etc which often have complex structures, but it is challenging to identify and classify such entities. Machine learning methods like CRF, MEMM and SVM have been widely used for learning to recognize such entities from an annotated corpus. The identification of appropriate feature templates and the selection of the important feature values play a very important role in the success of these methods. In this paper, we provide a study on word clustering and selection based feature reduction approaches for named entity recognition using a maximum entropy classifier. The identification and selection of features are largely done automatically without using domain knowledge. The performance of the system is found to be superior to existing systems which do not use domain knowledge.

Towards identifying intervention arms in randomized controlled trials: Extracting coordinating constructions

Background: Large numbers of reports of randomized controlled trials (RCTs) are published each year, and it is becoming increasingly difficult for clinicians practicing evidence-based medicine to find answers to clinical questions. The automatic machine extraction of RCT experimental details, including design methodology and outcomes, could help clinicians and reviewers locate relevant studies more rapidly and easily. Aim: This paper investigates how the comparison of interventions is documented in the abstracts of published RCTs. The ultimate goal is to use automated text mining to locate each intervention arm of a trial. This preliminary work aims to identify coordinating constructions, which are prevalent in the expression of intervention comparisons. Methods and results: An analysis of the types of constructs that describe the allocation of intervention arms is conducted, revealing that the compared interventions are predominantly embedded in coordinating constructions. A method is developed for identifying the descriptions of the assignment of treatment arms in clinical trials, using a full sentence parser to locate coordinating constructions and a statistical classifier for labeling positive examples. Predicate-argument structures are used along with other linguistic features with a maximum entropy classifier. An F-score of 0.78 is obtained for labeling relevant coordinating constructions in an independent test set. Conclusions: The intervention arms of a randomized controlled trials can be identified by machine extraction incorporating syntactic features derived from full sentence parsing.

Integration of Multiple Classifiers for Chinese Semantic Dependency Analysis

Semantic Dependency Analysis (SDA) has extensive applications in Natural Language Processing (NLP). In this paper, an integration of multiple classifiers is presented for SDA of Chinese. A Naive Bayesian Classifier, a Decision Tree and a Maximum Entropy classifier are used in a majority wins voting scheme. A portion of the Penn Chinese Treebank was manually annotated with semantic dependency structure. Then each of the three classifiers was trained on the same training data. All three of the classifiers were used to produce candidate relations for test data and the candidate relation that had the majority vote was chosen. The proposed approach achieved an accuracy of 86% in experimentation, which shows that the proposed approach is a promising one for semantic dependency analysis of Chinese.

Dialogue act recognition using maximum entropy

AbstractA dialogue‐based interface for information systems is considered a potentially very useful approach to information access. A key step in computer processing of natural‐language dialogues is dialogue‐act (DA) recognition. In this paper, we apply a feature‐based classification approach for DA recognition, by using the maximum entropy (ME) method to build a classifier for labeling utterances with DA tags. The ME method has the advantage that a large number of heterogeneous features can be flexibly combined in one classifier, which can facilitate feature selection. A unique characteristic of our approach is that it does not need to model the prior probability of DAs directly, and thus avoids the use of a discourse grammar. This simplifies the implementation of the classifier and improves the efficiency of DA recognition, without sacrificing the classification accuracy. We evaluate the classifier using a large data set based on the Switchboard corpus. Encouraging performance is observed; the highest classification accuracy achieved is 75.03%. We also propose a heuristic to address the problem of sparseness of the data set. This problem has resulted in poor classification accuracies of some DA types that have very low occurrence frequencies in the data set. Preliminary evaluation shows that the method is effective in improving the macroaverage classification accuracy of the ME classifier.

Chinese semantic dependency analysis

Semantic analysis is a standard tool in the Natural Language Processing (NLP) toolbox with widespread applications. In this article, we look at tagging part of the Penn Chinese Treebank with semantic dependency. Then we take this tagged data to train a maximum entropy classifier to label the semantic relations between headwords and dependents to perform semantic analysis on Chinese sentences. The classifier was able to achieve an accuracy of over 84%. We then analyze the errors in classification to determine the problems and possible solutions for this type of semantic analysis.

An Extension of Iterative Scaling for Decision and Data Aggregation in Ensemble Classification

Improved iterative scaling (IIS) is an algorithm for learning maximum entropy (ME) joint and conditional probability models, consistent with specified constraints, that has found great utility in natural language processing and related applications. In most IIS work on classification, discrete-valued "feature functions" are considered, depending on the data observations and class label, with constraints measured based on frequency counts, taken over hard (0---1) training set instances. Here, we consider the case where the training (and test) set consist of instances of probability mass functions on the features, rather than hard feature values. IIS extends in a natural way for this case. This has applications (1) to ME classification on mixed discrete-continuous feature spaces and (2) to ME aggregation of soft classifier decisions in ensemble classification. Moreover, we combine these methods, yielding a method, with proved learning convergence, that jointly performs (soft) decision-level and feature-level fusion in making ensemble decisions. We demonstrate favorable comparisons against standard Adaboost.M1, input-dependent boosting, and other supervised combining methods, on data sets from the UC Irvine Machine Learning repository.