Performance Of Taggers Research Articles

Evaluating NLP models with written and spoken L2 samples

The use of natural language processing tools such as part-of-speech taggers and syntactic parsers are increasingly being used in studies of second language (L2) proficiency and development. However, relatively little work has focused on reporting on the accuracy of these tools or optimizing their performance in L2 contexts. While some studies reference the published overall accuracy of a particular tool or include a small-scale accuracy analysis, very few (if any) studies provide a comprehensive account of the performance of taggers and parsers across a range of written and spoken registers. In this study, we provide a large-scale accuracy analysis of popular taggers and parsers across L1 and L2 written and spoken texts, both when default and L2-optimized models are used. Accuracy is examined both at the feature level (e.g., identifying adjective-noun relationships) and the text level (e.g., mean mutualinformation scores). The results highlight the strength and weaknesses of these tools.

APPLICATION OF HYBRID APPROACH FOR WOLAITA LANGUAGE PART OF SPEECH TAGGING

The main purpose of this study is to develop part-of-speech tagger for Wolaita Language using hybrid approach. Part of speech tagger is one of the subtasks in NLP application which is important for other Natural Language Processing (NLP) applications, like parser, machine translator, speech recognizer and search engines. PoST is a process of tagging a corresponding part of speech tag for a word that tag defines how the word is used in a sentence. The PoST for Wolaita language is not enough yet to be used as one vital module in other natural language processing applications. In this study, the development of PoS tagger using hybrid approach that combines HMM and rule based approaches was conducted for Wolaita language. In general HMM model need large data to increase the performance and the rule based model learner rule based on the language features. The HMM tagger, tags the words based on the optimal path for a given sequence of words and transformation based learning (TBL) is a rule based approaches that learns rule directly from the training corpus without expert knowledge. The developed hybrid approach of Wolaita language PoS tagger uses HMM tagger as initial annotators and rule based tagger as a corrector based on fixed threshold values. For implementation and experiment purpose the researcher used python programming and NLTK. For training and testing the models, 1256 sentences or 15,268 words are collected from three different categories (Bible, Social media in Wolaita language (Wogetta FM 96.6 ) and Wolaita language department) and annotated data manually. For tagging purpose 26 PoS tag are identified. From entire corpus, 90% for training and the remaining of entire corpus for testing purpose. The performance of the taggers, are tested by using different experiments. After experiment the researcher found that the performance of HMM, rule based and hybrid taggers shows 88.14%, 93.19% and 94.82% respectively. Generally, hybrid approach showed the better performance to assigning part of speech tag for Wolaita language.

Exploring unsupervised top tagging using Bayesian inference

Recognizing hadronically decaying top-quark jets in a sample of jets, or even its total fraction in the sample, is an important step in many LHC searches for Standard Model and Beyond Standard Model physics as well. Although there exists outstanding top-tagger algorithms, their construction and their expected performance rely on Montecarlo simulations, which may induce potential biases. For these reasons we develop two simple unsupervised top-tagger algorithms based on performing Bayesian inference on a mixture model. In one of them we use as the observed variable a new geometrically-based observable \tilde{A}_{3}Ã3, and in the other we consider the more traditional \tau_{3}/\tau_{2}τ3/τ2NN-subjettiness ratio, which yields a better performance. As expected, we find that the unsupervised tagger performance is below existing supervised taggers, reaching expected Area Under Curve AUC \sim 0.80-0.81∼0.80−0.81 and accuracies of about 69% -− 75% in a full range of sample purity. However, these performances are more robust to possible biases in the Montecarlo that their supervised counterparts. Our findings are a step towards exploring and considering simpler and unbiased taggers.

A Grammatically and Structurally Based Part of Speech (POS) Tagger for Arabic Language

In this paper we report on an experimental syntactically and morphologically driven rule-based Arabic tagger. The tagger is developed using Arabic language grammatical rules and regulations. The tagger requires no pre-tagged text and is developed using a primitive set of lexicon items along with extensive grammatical and structural rules. It is tested and compared to Stanford tagger both in terms of accuracy and performance (speed). Obtained results are quite comparable to Stanford tagger performance with marginal difference favoring the developed tagger in accuracy with huge difference in terms of speed of execution. The newly developed tagger named MTE Tagger has been tested and evaluated. For the evaluation of its accuracy of tagging, a set of Arabic text was manually prepared and annotated. Compared to Stanford tagger, the MTE tagger performance was quite comparable. The developed tagger makes use of no pre-annotated datasets, except of some simple lexicon consisting of list of words representing closed word types like demonstrative nouns or pronouns or some particles. For the purpose of evaluation of the new tagger, it was run on multiple datasets and results were compared to those of Stanford tagger. In particular, both taggers (the MTE and the Stanford) were run on a set of 1226 sentences with close to 20,000 tokens that was human annotated and verified to serve as testbed. The results were very encouraging where in both test runs, the MTE tagger outperformed the Stanford tagger in terms of accuracy of 87.88% versus 86.67% for the Stanford tagger. In terms of speed of tagging and in comparison Stanford tagger, MTE Taggers’ performance was on average 1:50. More improved accuracy is possible in future work as the set of rules are further optimized, integrated and more of Arabic language properties such as end of word discretization are used.

Identifying hadronically-decaying vector bosons and top quarks in ATLAS

Hadronic decays of vector bosons and top quarks are increasingly important to the ATLAS physics program, both in measurements of the Standard Model and searches for new physics. At high energies, these decays are collimated into a single overlapping region of energy deposits in the detector, referred to as a jet. However, vector bosons and top quarks are hidden under an enormous background of other processes producing jets. The ATLAS experiment has employed boosted decision trees and deep neural networks to the challenging task of identifying hadronically-decaying vector bosons and top quarks and rejecting other jet backgrounds. These discriminants are becoming increasingly complex and using more advanced machine learning techniques. The methods currently used to tag these objects are described. In order to improve the tagger performance on the signal efficiency and background rejection, new in-situ techniques are applied, thus directly evaluating the agreement between data and simulation after applying an arbitrarily complex classifier. The precision obtained by applying the in-situ techniques is presented.

How to tag non-standard language: Normalisation versus domain adaptation for Slovene historical and user-generated texts

AbstractPart-of-speech (PoS) tagging of non-standard language with models developed for standard language is known to suffer from a significant decrease in accuracy. Two methods are typically used to improve it: word normalisation, which decreases the out-of-vocabulary rate of the PoS tagger, and domain adaptation where the tagger is made aware of the non-standard language variation, either through supervision via non-standard data being added to the tagger’s training set, or via distributional information calculated from raw texts. This paper investigates the two approaches, normalisation and domain adaptation, on carefully constructed data sets encompassing historical and user-generated Slovene texts, in particular focusing on the amount of labour necessary to produce the manually annotated data sets for each approach and comparing the resulting PoS accuracy. We give quantitative as well as qualitative analyses of the tagger performance in various settings, showing that on our data set closed and open class words exhibit significantly different behaviours, and that even small inconsistencies in the PoS tags in the data have an impact on the accuracy. We also show that to improve tagging accuracy, it is best to concentrate on obtaining manually annotated normalisation training data for short annotation campaigns, while manually producing in-domain training sets for PoS tagging is better when a more substantial annotation campaign can be undertaken. Finally, unsupervised adaptation via Brown clustering is similarly useful regardless of the size of the training data available, but improvements tend to be bigger when adaptation is performed via in-domain tagging data.

From 0 to 10 million annotated words: part-of-speech tagging for Middle High German

By building a part-of-speech (POS) tagger for Middle High German, we investigate strategies for dealing with a low resource, diverse and non-standard language in the domain of natural language processing. We highlight various aspects such as the data quantity needed for training and the influence of data quality on tagger performance. Since the lack of annotated resources poses a problem for training a tagger, we exemplify how existing resources can be adapted fruitfully to serve as additional training data. The resulting POS model achieves a tagging accuracy of about 91% on a diverse test set representing the different genres, time periods and varieties of MHG. In order to verify its general applicability, we evaluate the performance on different genres, authors and varieties of MHG, separately. We explore self-learning techniques which yield the advantage that unannotated data can be utilized to improve tagging performance on specific subcorpora.

Top tagging: an analytical perspective

In this paper we study aspects of top tagging from first principles of QCD. We find that the method known as the CMS top tagger becomes collinear unsafe at high pt and propose variants thereof which are IRC safe, and hence suitable for analytical studies, while giving a comparable performance to the CMS tagger. We also develop new techniques to identify three-pronged jet substructure, based on adaptations of the Y-splitter method and its combination with grooming. A novel feature of our studies, relative to previous calculations of two-pronged substructure, is our use of triple-collinear splitting functions combined with all-order resummation, which owes to the presence of two mass scales of the same order, mt and mW, in the signal jet. We carry out leading logarithmic resummed calculations for the various top-taggers, for both background and signal jets, and compare the results to those from parton showers. We also identify and comment on the main features driving tagger performance at high pt and discuss the role of non-perturbative effects.

A new method to distinguish hadronically decaying boosted Z bosons from W bosons using the ATLAS detector.

The distribution of particles inside hadronic jets produced in the decay of boosted W and Z bosons can be used to discriminate such jets from the continuum background. Given that a jet has been identified as likely resulting from the hadronic decay of a boosted W or Z boson, this paper presents a technique for further differentiating Z bosons from W bosons. The variables used are jet mass, jet charge, and a b-tagging discriminant. A likelihood tagger is constructed from these variables and tested in the simulation of W'rightarrow WZ for bosons in the transverse momentum range 200 GeV <p_text {T}< 400 GeV in sqrt{s}=8 TeV pp collisions with the ATLAS detector at the LHC. For Z-boson tagging efficiencies of epsilon _Z=90, 50, and 10,%, one can achieve W^+-boson tagging rejection factors (1/epsilon _{W^+}) of 1.7, 8.3 and 1000, respectively. It is not possible to measure these efficiencies in the data due to the lack of a pure sample of high p_text {T}, hadronically decaying Z bosons. However, the modelling of the tagger inputs for boosted W bosons is studied in data using a tbar{t}-enriched sample of events in 20.3 fb{}^{-1} of data at sqrt{s}=8 TeV. The inputs are well modelled within uncertainties, which builds confidence in the expected tagger performance.

Internet Corpora: A Challenge for Linguistic Processing

Natural language processing tools are mostly developed for and optimized on newspaper texts, and often show a substantial performance drop when applied to other types of texts such as Twitter feeds, chat data or Internet forum posts. We explore a range of easy-to-implement methods of adapting existing part-of-speech taggers to improve their performance on Internet texts. Our results show that these methods can improve tagger performance substantially.

A token centric part-of-speech tagger for biomedical text.

Difficulties with part-of-speech (POS) tagging of biomedical text is accessing and annotating appropriate training corpora. These difficulties may result in POS taggers trained on corpora that differ from the tagger's target biomedical text (cross-domain tagging). In such cases where training and target corpora differ tagging accuracy decreases. This paper presents a POS tagger for cross-domain tagging called TcT. TcT estimates a tag's likelihood for a given token by combining token collocation probabilities and the token's tag probabilities calculated using a Naive Bayes classifier. We compared TcT to three POS taggers used in the biomedical domain (mxpost, Brill and TnT). We trained each tagger on a non-biomedical corpus and evaluated it on biomedical corpora. TcT was more accurate in cross-domain tagging than mxpost, Brill and TnT (respective averages 83.9, 81.0, 79.5 and 78.8). Our analysis of tagger performance suggests that lexical differences between corpora have more effect on tagging accuracy than originally considered by previous research work. Biomedical POS tagging algorithms may be modified to improve their cross-domain tagging accuracy without requiring extra training or large training data sets. Future work should reexamine POS tagging methods for biomedical text. This differs from the work to date that has focused on retraining existing POS taggers.

Using machine learning for concept extraction on clinical documents from multiple data sources

Concept extraction is a process to identify phrases referring to concepts of interests in unstructured text. It is a critical component in automated text processing. We investigate the performance of machine learning taggers for clinical concept extraction, particularly the portability of taggers across documents from multiple data sources. We used BioTagger-GM to train machine learning taggers, which we originally developed for the detection of gene/protein names in the biology domain. Trained taggers were evaluated using the annotated clinical documents made available in the 2010 i2b2/VA Challenge workshop, consisting of documents from four data sources. As expected, performance of a tagger trained on one data source degraded when evaluated on another source, but the degradation of the performance varied depending on data sources. A tagger trained on multiple data sources was robust, and it achieved an F score as high as 0.890 on one data source. The results also suggest that performance of machine learning taggers is likely to improve if more annotated documents are available for training. Our study shows how the performance of machine learning taggers is degraded when they are ported across clinical documents from different sources. The portability of taggers can be enhanced by training on datasets from multiple sources. The study also shows that BioTagger-GM can be easily extended to detect clinical concept mentions with good performance.

Persian POS tagging using probabilistic morphological analysis

Part of speech (POS) tagging as a fundamental task in natural language processing (NLP) has attracted many research efforts and many taggers are developed with different approaches to reach high performance and accuracy. In many complex applications of NLP, tagged corpora are among essential resources and designing an algorithm to create or enrich these resources is of high importance. Handling unknown words is a challenge in POS tagging which usually decreases the performance of taggers. This paper presents a POS tagger for Persian. It exploits a hybrid approach which is a combination of statistical and rule-based methods to tag Persian sentences. The proposed tagger uses a novel probabilistic morphological analysis to tag unknown words. As a secondary result of this research a knowledge base of Persian morphological rules with their probabilities is built according to a corpus. Experimental results show that our method improves the tagging performance and accuracy.