Maximum Entropy Classifier Research Articles

On the effects of using word2vec representations in neural networks for dialogue act recognition

Dialogue act recognition is an important component of a large number of natural language processing pipelines. Many research works have been carried out in this area, but relatively few investigate deep neural networks and word embeddings. This is surprising, given that both of these techniques have proven exceptionally good in most other language-related domains. We propose in this work a new deep neural network that explores recurrent models to capture word sequences within sentences, and further study the impact of pretrained word embeddings. We validate this model on three languages: English, French and Czech. The performance of the proposed approach is consistent across these languages and it is comparable to the state-of-the-art results in English. More importantly, we confirm that deep neural networks indeed outperform a Maximum Entropy classifier, which was expected. However, and this is more surprising, we also found that standard word2vec embeddings do not seem to bring valuable information for this task and the proposed model, whatever the size of the training corpus is. We thus further analyse the resulting embeddings and conclude that a possible explanation may be related to the mismatch between the type of lexical-semantic information captured by the word2vec embeddings, and the kind of relations between words that is the most useful for the dialogue act recognition task.

Dropped personal pronoun recovery in Chinese SMS

AbstractIn written Chinese, personal pronouns are commonly dropped when they can be inferred from context. This practice is particularly common in informal genres like Short Message Service messages sent via cell phones. Restoring dropped personal pronouns can be a useful preprocessing step for information extraction. Dropped personal pronoun recovery can be divided into two subtasks: (1) detecting dropped personal pronoun slots and (2) determining the identity of the pronoun for each slot. We address a simpler version of restoring dropped personal pronouns wherein only the person numbers are identified. After applying a word segmenter, we used a linear-chain conditional random field to predict which words were at the start of an independent clause. Then, using the independent clause start information, as well as lexical and syntactic information, we applied a conditional random field or a maximum-entropy classifier to predict whether a dropped personal pronoun immediately preceded each word and, if so, the person number of the dropped pronoun. We conducted a series of experiments using a manually annotated corpus of Chinese Short Message Service. Our approaches substantially outperformed a rule-based approach based partially on rules developed by Chung and Gildea (2010, Effects of Empty Categories on Machine Translation.Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics. pp. 636–45). Our approaches also outperformed (though by a considerably smaller margin) a machine-learning approach based closely on work by Yang, Liu, and Xue in (2015, Recovering Dropped Pronouns from Chinese Text Messages.Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics (ACL). Association for Computational Linguistics. pp. 309–13). Features derived from parsing largely did not help our approaches. We conclude that, given independent clause start information, the parse information we used was largely superfluous for identifying dropped personal pronouns.

Paraphrase identification and semantic text similarity analysis in Arabic news tweets using lexical, syntactic, and semantic features

The rapid growth in digital information has raised considerable challenges in particular when it comes to automated content analysis. Social media such as twitter share a lot of its users’ information about their events, opinions, personalities, etc. Paraphrase Identification (PI) is concerned with recognizing whether two texts have the same/similar meaning, whereas the Semantic Text Similarity (STS) is concerned with the degree of that similarity. This research proposes a state-of-the-art approach for paraphrase identification and semantic text similarity analysis in Arabic news tweets. The approach adopts several phases of text processing, features extraction and text classification. Lexical, syntactic, and semantic features are extracted to overcome the weakness and limitations of the current technologies in solving these tasks for the Arabic language. Maximum Entropy (MaxEnt) and Support Vector Regression (SVR) classifiers are trained using these features and are evaluated using a dataset prepared for this research. The experimentation results show that the approach achieves good results in comparison to the baseline results.

Twitter Influenza Surveillance: Quantifying Seasonal Misdiagnosis Patterns and their Impact on Surveillance Estimates.

BackgroundInfluenza (flu) surveillance using Twitter data can potentially save lives and increase efficiency by providing governments and healthcare organizations with greater situational awareness. However, research is needed to determine the impact of Twitter users’ misdiagnoses on surveillance estimates.ObjectiveThis study establishes the importance of Twitter users’ misdiagnoses by showing that Twitter flu surveillance in the United States failed during the 2011-2012 flu season, estimates the extent of misdiagnoses, and tests several methods for reducing the adverse effects of misdiagnoses.MethodsMetrics representing flu prevalence, seasonal misdiagnosis patterns, diagnosis uncertainty, flu symptoms, and noise were produced using Twitter data in conjunction with OpenSextant for geo-inferencing, and a maximum entropy classifier for identifying tweets related to illness. These metrics were tested for correlations with World Health Organization (WHO) positive specimen counts of flu from 2011 to 2014.ResultsTwitter flu surveillance erroneously indicated a typical flu season during 2011-2012, even though the flu season peaked three months late, and erroneously indicated plateaus of flu tweets before the 2012-2013 and 2013-2014 flu seasons. Enhancements based on estimates of misdiagnoses removed the erroneous plateaus and increased the Pearson correlation coefficients by .04 and .23, but failed to correct the 2011-2012 flu season estimate. A rough estimate indicates that approximately 40% of flu tweets reflected misdiagnoses.ConclusionsFurther research into factors affecting Twitter users’ misdiagnoses, in conjunction with data from additional atypical flu seasons, is needed to enable Twitter flu surveillance systems to produce reliable estimates during atypical flu seasons.

Assessing Quality of Care and Elder Abuse in Nursing Homes via Google Reviews.

BackgroundIt is challenging to assess the quality of care and detect elder abuse in nursing homes, since patients may be incapable of reporting quality issues or abuse themselves, and resources for sending inspectors are limited.ObjectiveThis study correlates Google reviews of nursing homes with Centers for Medicare and Medicaid Services (CMS) inspection results in the Nursing Home Compare (NHC) data set, to quantify the extent to which the reviews reflect the quality of care and the presence of elder abuse.MethodsA total of 16,160 reviews were collected, spanning 7,170 nursing homes. Two approaches were tested: using the average rating as an overall estimate of the quality of care at a nursing home, and using the average scores from a maximum entropy classifier trained to recognize indications of elder abuse.ResultsThe classifier achieved an F-measure of 0.81, with precision 0.74 and recall 0.89. The correlation for the classifier is weak but statistically significant: = 0.13, P < .001, and 95% confidence interval (0.10, 0.16). The correlation for the ratings exhibits a slightly higher correlation: = 0.15, P < .001. Both the classifier and rating correlations approach approximately 0.65 when the effective average number of reviews per provider is increased by aggregating similar providers.ConclusionsThese results indicate that an analysis of Google reviews of nursing homes can be used to detect indications of elder abuse with high precision and to assess the quality of care, but only when a sufficient number of reviews are available.

A Value Classification of Electronic Product Reviews Based on Maximum Entropy

Reviews of electronic product are important resources for manufacturers to gather feedbacks from customers. They provide an important reference for customers to make a decision on product purchase. Electronic product reviews with high quality is valuable, and reviews analysis is becoming an open issue. Based on the binary feature function, a maximum entropy classification of electronic product reviews is proposed to identify the valuable reviews. Considering the simpleness of the theme in reviews, a text classifier integrated with the features of descriptive anaphor is provided to enhance the model's fitting degree. Furthermore, the feature library is constructed by the Term frequency-inverse document frequency (TF-IDF) algorithm to locate the description, and the descriptive anaphors are identified by means of the minimum Inflectional phrase (IP) parsing tree. The experiment shows the proposed classifier has the relatively high stability and precision, which makes the reviews much more usable for manufacturers to make product strategy and for customers to make a decision.

Temporal replication of the national land cover database using active machine learning

Regional and national level land cover datasets, such as the National Land Cover Database (NLCD) in the United States, have become an important resource in physical and social science research. Updates to the NLCD have been conducted every 5 years since 2001; however, the procedure for producing a new release is labor-intensive and time-consuming, taking 3 or 4 years to complete. Furthermore, in most countries very few, if any, such releases exist, and thus there is high demand for efficient production of land cover data at different points in time. In this paper, an active machine learning framework for temporal updating (or backcasting) of land cover data is proposed and tested for three study sites covered by the NLCD. The approach employs a maximum entropy classifier to extract information from one Landsat image using the NLCD, and then replicate the classification on a Landsat image for the same geographic extent from a different point in time to create land cover data of similar quality. Results show that this framework can effectively replicate the land cover database in the temporal domain with similar levels of overall and within class agreement when compared against high resolution reference land cover datasets. These results demonstrate that the land cover information encapsulated in the NLCD can effectively be extracted using solely Landsat imagery for replication purposes. The algorithm is fully automated and scalable for applications at landscape and regional scales for multiple points in time.

Extending the geographic extent of existing land cover data using active machine learning and covariate shift corrective sampling

ABSTRACTConsistent land cover data provided at national and regional scales are increasingly relevant for a wide range of research topics from landscape ecology to population dynamics. As one example, the National Land Cover Database (NLCD) provides a valuable resource for research conducted at broad geographic scales across the US where survey-based land cover data are not available. However, the national extent of the NLCD (and similar databases produced in other countries) prevents studies from reaching across borders and thus limits potential applications at broader (e.g. multinational) scales. This article presents a framework for automated spatial extrapolation of a national land cover database, such as the NLCD using Landsat imagery alone. The extrapolation of high quality land cover data represents a unique opportunity to efficiently generate similar quality data for regions not originally covered. Extending the NLCD in the spatial domain based on remote-sensing imagery alone manifests itself as a domain adaptation challenge know as covariate shift, where the distribution of spectral information for the target data does not follow that of the source data. To overcome this problem, the algorithm implements a novel corrective sampling technique that facilitates the spatial extrapolation of land cover data. Using the corrected sample, an active machine learning routine was implemented with a maximum entropy classifier to replicate the NLCD for a different geographic extent. This framework was tested in three study sites to assess stability under different landscape conditions and the overall generalizability of the approach. Results produced similar levels of overall agreement as the NLCD when compared against reference datasets, showing that the NLCD can effectively be extended to new geographic extents using the proposed framework.

Combining Naïve Bayes and Modified Maximum Entropy Classifiers for Text Classification

Text Classificat ion is done mainly through classifiers proposed over the years, Naive Bayes and Maximum Entropy being the most popular of all. However, the individual classifiers show limited applicability according to their respective domains and scopes. Recent research works evaluated that the combination of classifiers when used for classification showed better performance than the individual ones. This work introduces a modified Maximum Entropy-based classifier. Maximum Entropy classifiers provide a great deal of flexibility for parameter defin itions and follow assumptions closer to real world scenario. This classifier is then combined with a Naive Bayes classifier. Naive Bayes Classification is a very simple and fast technique. The assumption model is opposite to that of Maximum Entropy. The combination of classifiers is done through operators that linearly co mbine the results of two classifiers to predict class of documents in query. Proper validation of the 7 proposed modifications (4 modifications of Maximum Entropy, 3 combined classifiers) are demonstrated through implementation and experimenting on real life datasets.

AN EFFECTIVE APPROACH FOR TEXT CLASSIFICATION

Text Classification is done mainly through classifiers proposed over the years, Naive Bayes and Maximum Entropy being the most popular of all. However, the individual classifiers show limited applicability according to their respective domains and scopes. Recent research works evaluated that the combination of classifiers when used for classification showed better performance than the individual ones. This work introduces a modified Maximum Entropy-based classifier. Maximum Entropy classifiers provide a great deal of flexibility for parameter definitions and follow assumptions closer to real world scenario. This classifier is then combined with a Naive Bayes classifier. Naive Bayes Classification is a very simple and fast technique. The assumption model is opposite to that of Maximum Entropy. The combination of classifiers is done through operators that linearly combine the results of two classifiers to predict class of documents in query. Proper validation of the 7 proposed modifications (4 modifications of Maximum Entropy, 3 combined classifiers) are demonstrated through implementation and experimenting on real life datasets.

Automated Classification of Selected Data Elements from Free-text Diagnostic Reports for Clinical Research.

In the Multiple Myeloma clinical registry at Heidelberg University Hospital, most data are extracted from discharge letters. Our aim was to analyze if it is possible to make the manual documentation process more efficient by using methods of natural language processing for multiclass classification of free-text diagnostic reports to automatically document the diagnosis and state of disease of myeloma patients. The first objective was to create a corpus consisting of free-text diagnosis paragraphs of patients with multiple myeloma from German diagnostic reports, and its manual annotation of relevant data elements by documentation specialists. The second objective was to construct and evaluate a framework using different NLP methods to enable automatic multiclass classification of relevant data elements from free-text diagnostic reports. The main diagnoses paragraph was extracted from the clinical report of one third randomly selected patients of the multiple myeloma research database from Heidelberg University Hospital (in total 737 selected patients). An EDC system was setup and two data entry specialists performed independently a manual documentation of at least nine specific data elements for multiple myeloma characterization. Both data entries were compared and assessed by a third specialist and an annotated text corpus was created. A framework was constructed, consisting of a self-developed package to split multiple diagnosis sequences into several subsequences, four different preprocessing steps to normalize the input data and two classifiers: a maximum entropy classifier (MEC) and a support vector machine (SVM). In total 15 different pipelines were examined and assessed by a ten-fold cross-validation, reiterated 100 times. For quality indication the average error rate and the average F1-score were conducted. For significance testing the approximate randomization test was used. The created annotated corpus consists of 737 different diagnoses paragraphs with a total number of 865 coded diagnosis. The dataset is publicly available in the supplementary online files for training and testing of further NLP methods. Both classifiers showed low average error rates (MEC: 1.05; SVM: 0.84) and high F1-scores (MEC: 0.89; SVM: 0.92). However the results varied widely depending on the classified data element. Preprocessing methods increased this effect and had significant impact on the classification, both positive and negative. The automatic diagnosis splitter increased the average error rate significantly, even if the F1-score decreased only slightly. The low average error rates and high average F1-scores of each pipeline demonstrate the suitability of the investigated NPL methods. However, it was also shown that there is no best practice for an automatic classification of data elements from free-text diagnostic reports.

Simulated and Self-Sustained Classification of Twitter Data based on its Sentiment

We present a methodology for naturally grouping the estimation of Twitter messages. Miniaturized scale websites are a testing new wellspring of data for information mining methods. The aim of this paper is to focus the careful feeling of the information from the microblogging site Twitter. Tweets regularly likewise contain URLs to different sites. Tweets additionally contain a certain measure of OOV (Out-Of-Vocabulary) words, for example, Hash tags, a labeling framework for points permitting Tweets in a comparative vein of discussion to be found. Other OOV words incorporate notice which is a system to direct a Tweet to one or more users. The KH coder tool gives a conventional precision result where the content is POS labeled and MySQL is utilized for putting away points of interest as a part of the database. The R tool is utilized to view the factual examination of information. Further, machine learning calculation has likewise been performed. A preprocessing and highlight choice system in blend with a Maximum Entropy, Naive Bayes and Decision Tree classifiers has been exhibited and sensible results has been delivered. Accuracy of the machine adapting methods for sentiment has been thought about and statistical representation of the classes has been depicted through KH Coder.

Combining Overall and Target Oriented Sentiment Analysis over Portuguese Text from Social Media

This document describes an approach to perform sentiment analysis on social media Portuguese content. In a single system, we perform polarity classification for both the overall sentiment, and target oriented sentiment. In both modes we train a Maximum Entropy classifier. The overall model is based on BoW type features, and also features derived from POS tagging and from sentiment lexicons. Target oriented analysis begins with named entity recognition, followed by the classification of sentiment polarity on these entities. This classifier model uses features dedicated to the entity mention textual zone, including negation detection, and the syntactic function of the target occurrence segment. Our experiments have achieved an accuracy of 75% for target oriented polarity classification, and 97% in overall polarity.

The contribution of co-reference resolution to supervised relation detection between bacteria and biotopes entities.

BackgroundThe acquisition of knowledge about relations between bacteria and their locations (habitats and geographical locations) in short texts about bacteria, as defined in the BioNLP-ST 2013 Bacteria Biotope task, depends on the detection of co-reference links between mentions of entities of each of these three types. To our knowledge, no participant in this task has investigated this aspect of the situation. The present work specifically addresses issues raised by this situation: (i) how to detect these co-reference links and associated co-reference chains; (ii) how to use them to prepare positive and negative examples to train a supervised system for the detection of relations between entity mentions; (iii) what context around which entity mentions contributes to relation detection when co-reference chains are provided.ResultsWe present experiments and results obtained both with gold entity mentions (task 2 of BioNLP-ST 2013) and with automatically detected entity mentions (end-to-end system, in task 3 of BioNLP-ST 2013). Our supervised mention detection system uses a linear chain Conditional Random Fields classifier, and our relation detection system relies on a Logistic Regression (aka Maximum Entropy) classifier. They use a set of morphological, morphosyntactic and semantic features. To minimize false inferences, co-reference resolution applies a set of heuristic rules designed to optimize precision. They take into account the types of the detected entity mentions, and take advantage of the didactic nature of the texts of the corpus, where a large proportion of bacteria naming is fairly explicit (although natural referring expressions such as "the bacteria" are common). The resulting system achieved a 0.495 F-measure on the official test set when taking as input the gold entity mentions, and a 0.351 F-measure when taking as input entity mentions predicted by our CRF system, both of which are above the best BioNLP-ST 2013 participant system.ConclusionsWe show that co-reference resolution substantially improves over a baseline system which does not use co-reference information: about 3.5 F-measure points on the test corpus for the end-to-end system (5.5 points on the development corpus) and 7 F-measure points on both development and test corpora when gold mentions are used. While this outperforms the best published system on the BioNLP-ST 2013 Bacteria Biotope dataset, we consider that it provides mostly a stronger baseline from which more work can be started. We also emphasize the importance and difficulty of designing a comprehensive gold standard co-reference annotation, which we explain is a key point to further progress on the task.

Topic-Based Coherence Modeling for Statistical Machine Translation

Coherence that ties sentences of a text into a meaningfully connected structure is of great importance to text generation and translation. In this paper, we propose topic-based coherence models to produce coherence for document translation, in terms of the continuity of sentence topics in a text. We automatically extract a coherence chain for each source text to be translated. Based on the extracted source coherence chain, we adopt a maximum entropy classifier to predict the target coherence chain that defines a linear topic structure for the target document. We build two topic-based coherence models on the predicted target coherence chain: 1) a word level coherence model that helps the decoder select coherent word translations and 2) a phrase level coherence model that guides the decoder to select coherent phrase translations. We integrate the two models into a state-of-the-art phrase-based machine translation system. Experiments on large-scale training data show that our coherence models achieve substantial improvements over both the baseline and models that are built on either document topics or sentence topics obtained under the assumption of direct topic correspondence between the source and target side. Additionally, further evaluations on translation outputs suggest that target translations generated by our coherence models are more coherent and similar to reference translations than those generated by the baseline.

METSP: a maximum-entropy classifier based text mining tool for transporter-substrate identification with semistructured text.

The substrates of a transporter are not only useful for inferring function of the transporter, but also important to discover compound-compound interaction and to reconstruct metabolic pathway. Though plenty of data has been accumulated with the developing of new technologies such as in vitro transporter assays, the search for substrates of transporters is far from complete. In this article, we introduce METSP, a maximum-entropy classifier devoted to retrieve transporter-substrate pairs (TSPs) from semistructured text. Based on the high quality annotation from UniProt, METSP achieves high precision and recall in cross-validation experiments. When METSP is applied to 182,829 human transporter annotation sentences in UniProt, it identifies 3942 sentences with transporter and compound information. Finally, 1547 confidential human TSPs are identified for further manual curation, among which 58.37% pairs with novel substrates not annotated in public transporter databases. METSP is the first efficient tool to extract TSPs from semistructured annotation text in UniProt. This tool can help to determine the precise substrates and drugs of transporters, thus facilitating drug-target prediction, metabolic network reconstruction, and literature classification.

HPS: High precision stemmer

Research into unsupervised ways of stemming has resulted, in the past few years, in the development of methods that are reliable and perform well. Our approach further shifts the boundaries of the state of the art by providing more accurate stemming results. The idea of the approach consists in building a stemmer in two stages. In the first stage, a stemming algorithm based upon clustering, which exploits the lexical and semantic information of words, is used to prepare large-scale training data for the second-stage algorithm. The second-stage algorithm uses a maximum entropy classifier. The stemming-specific features help the classifier decide when and how to stem a particular word.In our research, we have pursued the goal of creating a multi-purpose stemming tool. Its design opens up possibilities of solving non-traditional tasks such as approximating lemmas or improving language modeling. However, we still aim at very good results in the traditional task of information retrieval. The conducted tests reveal exceptional performance in all the above mentioned tasks. Our stemming method is compared with three state-of-the-art statistical algorithms and one rule-based algorithm. We used corpora in the Czech, Slovak, Polish, Hungarian, Spanish and English languages. In the tests, our algorithm excels in stemming previously unseen words (the words that are not present in the training set). Moreover, it was discovered that our approach demands very little text data for training when compared with competing unsupervised algorithms.

Combining Probabilistic Classifiers for Text Classification

Probabilistic classifiers are considered to be among the most popular classifiers for the machine learning community and are used in many applications. Although popular probabilistic classifiers exhibit very good performance when used individually in a specific classification task, very little work has been done on assessing the performance of two or more classifiers used in combination in the same classification task. In this work, we classify documents using two probabilistic approaches: The naive Bayes classifier and the Maximum Entropy classification model. Then, we combine the results of the two classifiers to improve the classification performance, using two merging operators, Max and Harmonic Mean. The proposed method was evaluated using the “ModApte” split of the Reuters-21578 dataset and the evaluation results show a measurable improvement in the final evaluation accuracy.

Identifying Abbreviations in Biomedical Literature Based on Maximum Entropy with Web Features

The number of biomedical literatures is growing rapidly, and biomedical literature mining is becoming essential. A learning classifier based on maximum entropy (ME) for identifying abbreviations is proposed. Two innovative Web-based features for extracting additional semantic information are developed. The study shows the Web as a knowledge source can be incorporated effectively in the machine learning framework and significantly improves its performance. The ME classifier achieves 95% precision and 89% recall on the gold standard corpus “Medstract” and 91% precision and 84% recall on the larger test data that includes 128 full text literatures.

Learning to Rank for Review Rating Prediction

The Internet has experienced profound changes. Large amount of user-generated-contents provide valuable information to the public. Customers usually express their opinion in online shopping. After they finish the reviews, they give an overall rating to the product or service. In this paper, we focus on the review rating prediction problem. Previous studies usually regard this problem as a regression problem. We take a new machine learning method to solve the problem. Learning to rank method has been exploited to tackle the prediction. After feature selection, the maximum entropy classifier has been employed to solve the multi-classification problem. The real life dataset has been crawled to verify the proposed method. Empirical studies demonstrate the proposed method outperform the baseline methods.