N-gram Model Research Articles

NLP ‘RECIPES’ FOR TEXT CORPORA: APPROACHES TO COMPUTING THE PROBABILITY OF A SEQUENCE OF TOKENS

Investigation in the hybrid architectures for Natural Language Processing (NLP) requires overcoming complexity in various intellectual traditions pertaining to computer science, formal linguistics, logic, digital humanities, ethical issues and so on. NLP as a subfield of computer science and artificial intelligence is concerned with interactions between computers and human (natural) languages. It is used to apply machine learning algorithms to text (and speech) in order to create systems, such as: machine translation (converting from text in a source language to text in a target language), document summarization (converting from long texts into short texts), named entity recognition, predictive typing, et cetera. Undoubtedly, NLP phenomena have been implanted in our daily lives, for instance automatic Machine Translation (MT) is omnipresent in social media (or on the world wide web), virtual assistants (Siri, Cortana, Alexa, and so on) can recognize a natural voice or e-mail services use detection systems to filter out some spam messages. The purpose of this paper, however, is to outline the linguistic and NLP methods to textual processing. Therefore, the bag-of-n-grams concept will be discussed here as an approach to extract more details about the textual data in a string of a grouped words. The n-gram language model presented in this paper (that assigns probabilities to sequences of words in text corpora) is based on findings compiled in Sketch Engine, as well as samples of language data processed by means of NLTK library for Python. Why would one want to compute the probability of a word sequence? The answer is quite obvious – in various systems for performing tasks, the goal is to generate texts that are more fluent. Therefore, a particular component is required, which computes the probability of the output text. The idea is to collect information how frequently the n-grams occur in a large text corpus and use it to predict the next word. Counting the number of occurrences can also envisage certain drawbacks, for instance there are sometimes problems with sparsity or storage. Nonetheless, the language models and specific computing ‘recipes’ described in this paper can be used in many applications, such as machine translation, summarization, even dialogue systems, etc. Lastly, it has to be pointed out that this piece of writing is a part of an ongoing work tentatively termed as LADDER (Linguistic Analysis of Data in the Digital Era of Research) that touches upon the process of datacization[1] that might help to create an intelligent system of interdisciplinary information.

Aspect-Based Sentiment Analysis and Emotion Detection for Code-Mixed Review

Review can affect customer decision making because by reading it, people manage to know whether the review is positive, or negative. However, positive, negative, and neutral, without considering the emotion will be not enough because emotion can strengthen the sentiment result. This study explains about the comparison of machine learning and deep learning in sentiment as well as emotion classification with multi-label classification. In machine learning comparison, the problem transformation that we used are Binary Relevance (BR), Classifier Chain (CC), and Label Powerset (LP), with Decision Tree (DT), Random Forest (RF), Support Vector Machine (SVM), and Extra Tree Classifier (ET) as algorithms of machine learning. The features we compared are n-gram language model (unigram, bigram, unigram-bigram). For deep learning, algorithms that we applied are Gated Recurrent Unit (GRU) and Bidirectional Long Short-Term Memory (BiLSTM), using self-developed word embedding. The comparison results show RF dominates with 88.4% and 89.54% F1 scores with CC method for food aspect, and LP for price, respectively. For service and ambience aspects, ET leads with 92.65% and 87.1% with LP and CC methods, respectively. On the other hand, in deep learning comparison, GRU and BiLSTM obtained similar F1- score for food aspect, 88.16%. On price aspect, GRU leads with 83.01%. However, for service and ambience, BiLSTM achieved higher F1-score, 89.03% and 84.78%.

A Sentiment Analysis Method of Capsule Network Based on BiLSTM

Nowadays, capsule network model is widely used in image processing, whose feature engineering is not suitable for sentiment analysis based on texts obviously. In this paper, we propose a capsule network model with BiLSTM named caps-BiLSTM for sentiment analysis to solve the problem, and introduce the experimental results on different datasets. At the beginning of caps-BiLSTM, a convolution layer is used to transform the instance to hide vector. Then the capsule module constructs the capsule representation to the n-gram model. The state probability of a certain capsule is calculated by the capsule model. If the state probability of a given instance is the largest among all capsules, a higher coupling coefficient is assigned. Finally, in order to fusion the data features, the output of the capsule enters into a BiLSTM structure, which is used as a decoder to get the probability representation. Experimental results based on MR, IMDB and SST datasets show that the proposed method can achieve better performances than the traditional machine learning methods and the compared deeping learning models.

N-gram based Machine Translation for English-Assamese: Two Languages with High Syntactical Dissimilarity

To bridge the language constraint of the people residing in northeastern region of India, machine translation system is a necessity. Large number of people in this region cannot access many services due to the language incomprehensibility. Among several languages spoken, Assamese is one of the major languages used in northeast India. Machine translation for Assamese language is limited compared to other languages. As a result, large number of people using Assamese language cannot avail lots of benefits associated with it. This paper has focused on the development of the English to Assamese translation system using n-gram model. The n-gram model works very well with the language pair having high dissimilarity in syntax compared to other models. The value of n has a very big role in the quality and efficiency of the system. Bilingual Evaluation Understudy (BLEU) score differs significantly with the change of the n-gram. This model uses tuples to reduce the consumption of excess memory and to accelerate the translation process. Parallel corpus has been used for training the n-gram based decoder called MARIE. The number of translation units extracted using n-gram model is much less than the translation units extracted using phrase based model. This has a high impact on system efficiency.

Automated Misspelling Detection and Correction in Persian Clinical Text.

Accurate electronic health records are important for clinical care, research, and patient safety assurance. Correction of misspelled words is required to ensure the correct interpretation of medical records. In the Persian language, the lack of automated misspelling detection and correction system is evident in the medicine and health care. In this article, we describe the development of an automated misspelling detection and correction system for radiology and ultrasound's free texts in the Persian language. To achieve our goal, we used n-gram language model and three different types of free texts related to abdominal and pelvic ultrasound, head and neck ultrasound, and breast ultrasound reports. Our system achieved the detection performance of up to 90.29% for radiology and ultrasound's free texts with the correction accuracy of 88.56%. Results indicated that high-quality spelling correction is possible in clinical reports. The system also achieved significant savings during the documentation process and final approval of the reports in the imaging department.

I Say, You Say, We Say

Collaborative problem solving (CPS) is a crucial 21st century skill; however, current technologies fall short of effectively supporting CPS processes, especially for remote, computer-enabled interactions. In order to develop next-generation computer-supported collaborative systems that enhance CPS processes and outcomes by monitoring and responding to the unfolding collaboration, we investigate automated detection of three critical CPS process ? construction of shared knowledge, negotiation/coordination, and maintaining team function ? derived from a validated CPS framework. Our data consists of 32 triads who were tasked with collaboratively solving a challenging visual computer programming task for 20 minutes using commercial videoconferencing software. We used automatic speech recognition to generate transcripts of 11,163 utterances, which trained humans coded for evidence of the above three CPS processes using a set of behavioral indicators. We aimed to automate the trained human-raters' codes in a team-independent fashion (current study) in order to provide automatic real-time or offline feedback (future work). We used Random Forest classifiers trained on the words themselves (bag of n-grams) or with word categories (e.g., emotions, thinking styles, social constructs) from the Linguistic Inquiry Word Count (LIWC) tool. Despite imperfect automatic speech recognition, the n-gram models achieved AUROC (area under the receiver operating characteristic curve) scores of .85, .77, and .77 for construction of shared knowledge, negotiation/coordination, and maintaining team function, respectively; these reflect 70%, 54%, and 54% improvements over chance. The LIWC-category models achieved similar scores of .82, .74, and .73 (64%, 48%, and 46% improvement over chance). Further, the LIWC model-derived scores predicted CPS outcomes more similar to human codes, demonstrating predictive validity. We discuss embedding our models in collaborative interfaces for assessment and dynamic intervention aimed at improving CPS outcomes.

Athena: Automated Tuning of k-mer based Genomic Error Correction Algorithms using Language Models

The performance of most error-correction (EC) algorithms that operate on genomics reads is dependent on the proper choice of its configuration parameters, such as the value of k in k-mer based techniques. In this work, we target the problem of finding the best values of these configuration parameters to optimize error correction and consequently improve genome assembly. We perform this in an adaptive manner, adapted to different datasets and to EC tools, due to the observation that different configuration parameters are optimal for different datasets, i.e., from different platforms and species, and vary with the EC algorithm being applied. We use language modeling techniques from the Natural Language Processing (NLP) domain in our algorithmic suite, Athena, to automatically tune the performance-sensitive configuration parameters. Through the use of N-Gram and Recurrent Neural Network (RNN) language modeling, we validate the intuition that the EC performance can be computed quantitatively and efficiently using the “perplexity” metric, repurposed from NLP. After training the language model, we show that the perplexity metric calculated from a sample of the test (or production) data has a strong negative correlation with the quality of error correction of erroneous NGS reads. Therefore, we use the perplexity metric to guide a hill climbing-based search, converging toward the best configuration parameter value. Our approach is suitable for both de novo and comparative sequencing (resequencing), eliminating the need for a reference genome to serve as the ground truth. We find that Athena can automatically find the optimal value of k with a very high accuracy for 7 real datasets and using 3 different k-mer based EC algorithms, Lighter, Blue, and Racer. The inverse relation between the perplexity metric and alignment rate exists under all our tested conditions—for real and synthetic datasets, for all kinds of sequencing errors (insertion, deletion, and substitution), and for high and low error rates. The absolute value of that correlation is at least 73%. In our experiments, the best value of k found by Athena achieves an alignment rate within 0.53% of the oracle best value of k found through brute force searching (i.e., scanning through the entire range of k values). Athena’s selected value of k lies within the top-3 best k values using N-Gram models and the top-5 best k values using RNN models With best parameter selection by Athena, the assembly quality (NG50) is improved by a Geometric Mean of 4.72X across the 7 real datasets.

Language Identification for Multilingual Sentiment Examination

Social media is most popular platform on which users can share their views, reviews and knowledge about various topics, news, products etc. Identifying sentiments or opinions of users is valuable for many e-commerce companies, Hotels, e-learning etc. This opinion analysis is useful for companies to improve their service and products. Due to increase in web users across globe, users happen to post their views freely over the internet. Many different languages are spoken across globe, supporting multilingual nature of social media makes analysis of such text difficult. Sentiment analysis can be conducted using videos, image, text, where text sentiment analysis is most popular form because of freely available contents in the form of blogs, reviews, comments etc. Because of development of social media platform, people can post comment in any language, creates the need for Multilingual sentiment analysis. Sentiment analysis task needs phases such as data collection, pre-processing, sentiment classification and polarity identification. The Multilingual nature needs Script Identification on the input text by labelling the different words used in text along with scripts used to denote them. Various languages used in the text are identified and the Hindi language text written in Romanized script is transliterated to Devanagari script. Text is then completely translated into English language and POS(Parts of Speech) tagging is performed on the obtained text. The aim and purpose of this study is to survey different techniques of multilingual sentiment analysis, and language identification of source text, where n-grams model outperforms all.

MiNgMatch—A Fast N-gram Model for Word Segmentation of the Ainu Language

Word segmentation is an essential task in automatic language processing for languages where there are no explicit word boundary markers, or where space-delimited orthographic words are too coarse-grained. In this paper we introduce the MiNgMatch Segmenter—a fast word segmentation algorithm, which reduces the problem of identifying word boundaries to finding the shortest sequence of lexical n-grams matching the input text. In order to validate our method in a low-resource scenario involving extremely sparse data, we tested it with a small corpus of text in the critically endangered language of the Ainu people living in northern parts of Japan. Furthermore, we performed a series of experiments comparing our algorithm with systems utilizing state-of-the-art lexical n-gram-based language modelling techniques (namely, Stupid Backoff model and a model with modified Kneser-Ney smoothing), as well as a neural model performing word segmentation as character sequence labelling. The experimental results we obtained demonstrate the high performance of our algorithm, comparable with the other best-performing models. Given its low computational cost and competitive results, we believe that the proposed approach could be extended to other languages, and possibly also to other Natural Language Processing tasks, such as speech recognition.

Evaluating Computational Language Models with Scaling Properties of Natural Language

In this article, we evaluate computational models of natural language with respect to the universal statistical behaviors of natural language. Statistical mechanical analyses have revealed that natural language text is characterized by scaling properties, which quantify the global structure in the vocabulary population and the long memory of a text. We study whether five scaling properties (given by Zipf’s law, Heaps’ law, Ebeling’s method, Taylor’s law, and long-range correlation analysis) can serve for evaluation of computational models. Specifically, we test n-gram language models, a probabilistic context-free grammar, language models based on Simon/Pitman-Yor processes, neural language models, and generative adversarial networks for text generation. Our analysis reveals that language models based on recurrent neural networks with a gating mechanism (i.e., long short-term memory; a gated recurrent unit; and quasi-recurrent neural networks) are the only computational models that can reproduce the long memory behavior of natural language. Furthermore, through comparison with recently proposed model-based evaluation methods, we find that the exponent of Taylor’s law is a good indicator of model quality.

A Higher-Order N-gram Model to Enhance Automatic Word Prediction for Assamese Sentences Containing Ambiguous Words

Word prediction is a technique which tries to suggest the users’ words after knowing the few input letters of the user. This predictive model also tries to generate the future words or next words of a sentence by observing earlier words of the sentence. In this research, two problems are combined, one is word prediction and the next is handling of ambiguous words. A word prediction model predicts the future words of a sentence by using n-gram based model. In general, predictive models use unigram, bigram or trigram models to predict the next words. In case of sentences consisting of ambiguous words, the predictive model by using only bigram or trigram cannot perform well to predict the next words. To enhance this prediction for ambiguous words, maximum of six previous input words are observed and try to predict almost the exact words after the ambiguous words in those particular contexts. Different level of experiments are done and the results are compared for modified or enhanced prediction model with the traditional prediction model, improvement on accuracy and failure rate are found in the enhanced model. The accuracy of the Traditional Model is 60.68% on the hand the accuracy of the Enhanced Model is 66.88%. The failure rate of the Traditional Model is 32.35% and the Enhanced Model is 29.17%

A distributed system for large-scale n-gram language models at Tencent

n-gram language models are widely used in language processing applications, e.g., automatic speech recognition, for ranking the candidate word sequences generated from the generator model, e.g., the acoustic model. Large n-gram models typically give good ranking results; however, they require a huge amount of memory storage. While distributing the model across multiple nodes resolves the memory issue, it nonetheless incurs a great network communication overhead and introduces a different bottleneck. In this paper, we present our distributed system developed at Tencent with novel optimization techniques for reducing the network overhead, including distributed indexing, batching and caching. They reduce the network requests and accelerate the operation on each single node. We also propose a cascade fault-tolerance mechanism which adaptively switches to small n-gram models depending on the severity of the failure. Experimental study on 9 automatic speech recognition (ASR) datasets confirms that our distributed system scales to large models efficiently, effectively and robustly. We have successfully deployed it for Tencent's WeChat ASR with the peak network traffic at the scale of 100 millions of messages per minute.

A Chinese unknown word recognition method for micro-blog short text based on improved FP-growth

Unknown word recognition is one of the important research contents of natural language processing. However, there are still problems such as sparse data, corpus noise, and various forms of expressions for the identification of micro-blog short words. This paper proposes an unknown words recognition method POS-FP (Frequent Pattern growth with part- of-speech)for micro-blog short text. Firstly, the candidate unknown words are obtained by combing the N-grams model and frequent item sets. Then the unknown word is filtered and verified by the improved mutual information, information entropy and context dependence. Finally, the open verification method is used to obtain final unknown word. Experiments show that the algorithm improved the unknown word recognition for micro-blog short texts.

The effects and preventability of 2627 patient safety incidents related to health information technology failures: a retrospective analysis of 10 years of incident reporting in England and Wales

The use of health information technology (IT) is rapidly increasing to support improvements in the delivery of care. Although health IT is delivering huge benefits, new technology can also introduce unique risks. Despite these risks, evidence on the preventability and effects of health IT failures on patients is scarce. In our study we therefore sought to evaluate the preventability and effects of health IT failures by examining patient safety incidents in England and Wales. We designed our study as a retrospective analysis of 10 years of incident reporting in England and Wales. We used text mining with the words "computer", "system", "workstation", and "network" to explore free-text incident descriptors to identify incidents related to health IT failures following a previously described approach. We then applied an n-gram model of searching to identify contiguous sequences of words and provide spatial context. We examined incident details, recorded harm, and preventability. Standard descriptive statistics were applied. Degree of harm was identified according to standardised definitions and preventability was assessed by two independent reviewers. We identified 2627 incidents related to health IT failures. 2557 (97%) of 2627 incidents were assessed for harm (70 incidents were excluded). 2106 (82%) of 2557 health IT failures caused no harm to patients, 331 (13%) caused low harm, 102 (4%) caused moderate harm, 14 (1%) caused severe harm, and four (<1%) contributed to the death of a patient. 1964 (75%) of 2627 incidents were deemed to be preventable. Health IT is fundamental to the delivery of high-quality care, yet there is a poor understanding of the effects of IT failures on patient safety and whether they can be prevented. Failures are complex and involve interlinked aspects of technology, people, and the environment. Health IT failures are undoubtedly a potential source of substantial harm, but they are likely to be under-reported. Worryingly, three-quarters of IT failures are potentially preventable. There is a need to see health IT as a fundamental tenet of patient safety, develop better methods for capturing the effects of IT failures on patients, and adopt simple measures to reduce their probability and mitigate their risk. The National Institutes of Health Research Imperial Patient Safety Translational Research Centre at Imperial College London.

Recurrent neural network with attention mechanism for language model

The rapid growth of the Internet promotes the growth of textual data, and people get the information they need from the amount of textual data to solve problems. The textual data may include some potential information like the opinions of the crowd, the opinions of the product, or some market-relevant information. However, some problems that point to “How to get features from the text” must be solved. The model of extracting the text features by using the neural network method is called neural network language model. The features are based on n-gram Model concept, which are the co-occurrence relationship between the vocabularies. The word vectors are important because the sentence vectors or the document vectors still have to understand the relationship between the words, and based on this, this study discusses the word vectors. This study assumes that the words contain “the meaning in sentences” and “the position of grammar.” This study uses recurrent neural network with attention mechanism to establish a language model. This study uses Penn Treebank, WikiText-2, and NLPCC2017 text datasets. According to these datasets, the proposed models provide the better performance by the perplexity.

Literature Review of Sentiment Analysis Techniques for Microblogging Site

Any opinion/review given by any of an individual through which the feelings, text message, attitudes and thoughts can be expressed is known as sentiment. The kinds of data analysis which is attained from the news reports, user reviews, social media updates or microblogging sites is called sentiment analysis which is also known as opinion mining. It is an approach which is used to analyse sentiment of input data. The reviews of individuals towards certain events, brands, product or company can be known through sentiment analysis. The responses of general public are collected and improvised by researchers to perform evaluations. The popularity of sentiment analysis is growing today since the numbers of views being shared by people on the microblogging sites are also increasing. All the sentiments can be categorized into three different categories called positive, negative and neutral. Twitter, being the most popular microblogging site, is used to collect the data to perform analysis. Tweepyis used to extract the source data from Twitter. Python language is used in this research to implement the classification algorithm on the collected data. The features are extracted using N-gram modeling technique. The sentiments are categorized among positive, negative and neutral using a supervised machine learning algorithm known as K-Nearest Neighbor.

Hybrid N-gram model using Naïve Bayes for classification of political sentiments on Twitter

Twitter, an online micro-blogging and social networking service, provides registered users the ability to write in 140 characters anything they wish and hence providing them the opportunity to express their opinions and sentiments on events taking place. Politically sentimental tweets are top-trending tweets; whenever election is near, users tweet about their favorite candidates or political parties and at times give their reasons for that. In this study, we hybridize two n-gram [two n-gram models used in this study are unigram and n-gram. Therefore, in this study, where unigram is mentioned that refers to a least-order n-gram (unigram) and where n-gram is mentioned that refers to the highest-order (full sentence or tweet level) n-gram] models and applied Laplace smoothing to Naive Bayesian classifier and Katz back-off on the model. This was done in order to smoothen and address the limitation of accuracy in terms of precision and recall of n-gram models caused by the ‘zero count problem.’ Result from our baseline model shows an increase of 6.05% in average F-Harmonic accuracy in comparison with the n-gram model and 1.75% increase in comparison with the semantic-topic model proposed from a previous study on the same dataset, i.e., Obama–McCain dataset.

Automatic Dating of Medieval Charters from Denmark

Dating of medieval text sources is a central task common to the field of manuscript studies. It is a difficult process requiring expert philological and historical knowledge. We investigate the issue of automatic dating of a collection of about 300 charters from medieval Denmark, in particular how n-gram models based on different transcription levels of the charters can be used to assign the manuscripts to a specific temporal interval. We frame the problem as a classification task by dividing the period into bins of 50 years and using these as classes in a supervised learning setting to develop SVM classifiers. We show that the more detailed facsimile transcription, which captures palaeographic characteristics of a text, provides better results than the diplomatic level, where such distinctions are normalised. Furthermore, both character and word n-grams show promising results, the highest accuracy reaching 74.96 %. This level of classification accuracy corresponds to being able to date almost 75 % of the charters with a 25-year error margin, which philologists use as a standard of the precision with which medieval texts can be dated manually.

Melodic patterns and tonal cadences: Bayesian learning of cadential categories from contrapuntal information

ABSTRACTRecent work has shown that authentic and half cadences can be identified via harmonic features in both supervised and unsupervised settings, suggesting that humans may use such cues in perceiving and learning cadences. The present study tests melodic features in these same tasks. Both n-gram models and profile hidden Markov models of melodic patterns are used for supervised classification and unsupervised learning of cadences in Classical string quartets. Success is achieved at the supervised task but not the unsupervised task, indicating that melodic cues would help in perceiving cadences but not in learning to perceive them.

A Review towards the Sentiment Analysis Techniques for the Analysis of Twitter Data

Any opinion of an individual through which the feelings, attitudes and thoughts can be expressed is known as sentiment. The kinds of data analysis which is attained from the news reports, user reviews, social media updates or microblogging sites is called sentiment analysis which is also known as opinion mining. The reviews of individuals towards certain events, brands, product or company can be known through sentiment analysis. The responses of general public are collected and improvised by researchers to perform evaluations. The popularity of sentiment analysis is growing today since the numbers of views being shared by people on the microblogging sites are also increasing. All the sentiments can be categorized into three different categories called positive, negative and neutral. Twitter, being the most popular microblogging site, is used to collect the data to perform analysis. Tweepy is used to extract the source data from Twitter. Python language is used in this research to implement the classification algorithm on the collected data. The features are extracted using N-gram modeling technique. The sentiments are categorized among positive, negative and neutral using a supervised machine learning algorithm known as K-Nearest Neighbor.