Field Of NLP Research Articles

Graph Transformer for Communities Detection in Social Networks

Graphs are used in various disciplines such as telecommunication, biological networks, as well as social networks. In large-scale networks, it is challenging to detect the communities by learning the distinct properties of the graph. As deep learning has made contributions in a variety of domains, we try to use deep learning techniques to mine the knowledge from large-scale graph networks. In this paper, we aim to provide a strategy for detecting communities using deep autoencoders and obtain generic neural attention to graphs. The advantages of neural attention are widely seen in the field of NLP and computer vision, which has low computational complexity for large-scale graphs. The contributions of the paper are summarized as follows. Firstly, a transformer is utilized to downsample the first-order proximities of the graph into a latent space, which can result in the structural properties and eventually assist in detecting the communities. Secondly, the fine-tuning task is conducted by tuning variant hyperparameters cautiously, which is applied to multiple social networks (Facebook and Twitch). Furthermore, the objective function (cross-entropy) is tuned by L0 regularization. Lastly, the reconstructed model forms communities that present the relationship between the groups. The proposed robust model provides good generalization and is applicable to obtaining not only the community structures in social networks but also the node classification. The proposed graph-transformer shows advanced performance on the social networks with the average NMIs of 0.67 ± 0.04, 0.198 0.02, 0.228 ± 0.02, and 0.68 ± 0.03 on Wikipedia crocodiles, Github Developers, Twitch England, and Facebook Page-Page networks, respectively.

Machine Learning-Based Grammar Error Detection Method in English Composition

The detection of grammatical errors in English composition is an important task in the field of NLP. The main purpose of this task is to check out grammatical errors in English sentences and correct them. Grammatical error detection and correction are important applications in the automatic proofreading of English texts and in the field of English learning aids. With the increasing influence of English on a global scale, a huge breakthrough has been made in the task of detecting English grammatical errors. Based on machine learning, this paper designs a new method for detecting grammatical errors in English composition. First, this paper implements a grammatical error detection model based on Seq2Seq. Second, this paper implements a grammatical error detection and correction scheme based on the Transformer model. The Transformer model performs better than most grammar models. Third, this paper realizes the application of the BERT model in grammar error detection and error correction tasks, and the generalization ability of the model has been significantly enhanced. This solves the problem that the forward and backward cannot be merged when the Transformer trains the language model. Fourth, this paper proposes a method of grammatical error detection and correction in English composition based on a hybrid model. According to specific application scenarios, the corresponding neural network model is used for grammatical error correction. Combine the Seq2Seq structure to encode the input sequence and automate feature engineering. Through the combination of traditional model and deep model, the advantages are complemented to realize grammatical error detection and automatic correction.

Improved Word Sense Determination in Malayalam using Latent Dirichlet Allocation and Semantic Features

Recent years have witnessed phenomenal developments worldwide in the field of NLP . But developments in Indian regional languages are very few compared to them. This work is a step towards the construction of a target word sense disambiguation system in Malayalam, which is the regional language of the state of Kerala, India. Word Sense Disambiguation/Determination refers to the task of correctly identifying the sense of an ambiguous word from its context. This is considered an AI-Complete problem in the field of Natural Language Processing . For this purpose, an exclusive corpus of 1,147 contexts of target ambiguous words has been created, which to the best of our knowledge is the first attempt in Malayalam. This work describes how the performance of an unsupervised LDA-based approach towards WSD could be improved using semantic features like synonyms and co-occurrence information.

Word Sense Disambiguation and Semantics techniques

Of the many kinds of ambiguity in language is word sense. This problem had received the most attention in computational linguistics, and the reasons for this are clear: their resolution is seemingly essential for any practical application, and they seem to require a wide variety of methods and knowledge-sources with no pattern apparent in what any particular instance requires.Since the beginning of NLP field and the researchers trying to bypass the bottleneck of word sense disambiguation (WSD). The lack of high-performing methods for sense disambiguation may be considered the major obstacle that prevented an extensive use of natural language processing techniques in many areas of information technology, such as information classification and retrieval, query processing, advanced Web search, document warehousing, etc.This obstacle inspired many researchers to come up with new methods, approaches, and to exploit whatever knowledge resources exist to develop a solution. Previously most of the techniques used were based on a huge amount of knowledge sources, which suffers from the problems of incomplete, inadequate, and expensive resources.After that a trained techniques and the statistical approaches dominant over the field for a while were at the beginning good results obtained with limited and small domains. But with the expanding of the domains, the luck of good trained knowledge sources besides the neglecting of the semantic roles did not lay good results at the end.Now a day’s the research had been directed towards a hybrid approach that depends on the available and more reliable knowledge sources besides the semantic techniques, and training methods which consider the role of the words in its context.In this research we will take a case study of WSD. Then we will display the semantic techniques used in it and display the effect of implementing consistence concept evaluation method to produce more reliable results based on semantic similarity, besides that we will show how more conceptualization of the semantic relations can be implemented through semantic information.

TechWordNet: Development of semantic relation for technology information analysis using F-term and natural language processing

Text analysis on technology has recently been progressing from the level of words to semantic relations between words. However, existing research methods, such as Subject-Action-Object, have focused on specific purposes or analytical techniques. There is an insufficient amount of fundamental study on what types of semantic relations in technical information need to be analysed to provide meaningful information. At the same time, in the field of NLP, the deep learning-based semantic relation model has been establishing as useful for specific tasks. However, there is a limit to applying the NLP model itself for technical analysis because it does not consider the characteristics of the textual information about technology. Therefore, this study proposes a deep learning-based semantic relation model for technology information analysis. First, meaningful types of semantic relations are derived from the text information about technology. By analysing the F-term classification code, which is a multi-dimensional technology hierarchy with descriptions, a technology semantic labelled dataset is constructed. Finally, we develop a classification model that analyses the semantic relations of technology based on the sentence embedding model. This study contributes to the construction of a deep learning model by developing a meaningful type in the analysis of technical information and constructing a technical text dataset with labels. The result of semantic technology relations can also be utilized as a high-quality source for various applications on technology analysis, such as technology tree and technology roadmap. In other words, it has the advantage of being able to provide generalizable technical information that is not dependent on a specific analysis purpose.

Comparison of word embeddings in text classification based on RNN and CNN

This paper presents a comparison of word embeddings in text classification using RNN and CNN. In the field of image classification, deep learning methods like as RNN and CNN have shown to be popular. CNN is most popular model among deep learning techniques in the field of NLP because of its simplicity and parallelism, even if the dataset is huge. Word embedding techniques employed are GloVe and fastText. Use of different word embeddings showed a major difference in the accuracy of the models. When it comes to embedding of rare words, GloVe can sometime perform poorly. Inorder to tackle this issue, fastText method is used. Deep neural networks with fastText showed a remarkable improvement in the accuracy than GloVe. But fastText took some time to train when compared to GloVe. Further, the accuracy was improved by minimizing the batch size. Finally we concluded that the word embeddings have a huge impact on the performance of text classification models

Research of semantic role labeling based on long short-term memory neural networks

Semantic Role Labeling (SRL) is a shallow semantic analysis in the field of NLP, and a relatively basic and important step. Traditionally, SRL has been performed based on the results of syntactic analysis and has problems such as over-reliance on feature engineering. With the development of deep learning, many neural network models for NLP have been proposed and SRL tasks can be performed well by neural networks. Among these, long short-term memory networks form a very good fit with the SRL task by virtue of their sequence-based features. In this paper, in order, we first analyze the SRL task based on grammatical analysis and neural networks, then the SRL task based on LSTM and its improved models, then the dataset and model evaluation of the SRL task, and finally conclude and look forward.

АВТОМАТИЧНА ГЕНЕРАЦІЯ НАВЧАЛЬНИХ ТЕСТІВ ЗА ДОПОМОГОЮ ПРОГРАМНОЇ ОБРОБКИ ПРИРОДНО-МОВНИХ ТЕКСТІВ

У статті розглядаються основні технології, які сьогодні використовуються в електронному навчанні в багатьох вищих навчальних закладах. Згадано пакет програм SMPR, створений студентами факультету кібернетики та інформаційних технологій Київського національного університету імені Тараса Шевченка протягом декількох років. Їх метою є спрощення процесу навчання, консультування та перевірки знань учнів як для викладачів, так і для студентів. SMPR використовується для навчання викладачами кількох університетів для підтримки курсу «Теорія рішень» та інших предметів прикладних наук. У 2014 році цей програмний пакет удосконалено за допомогою автоматизованої підсистеми оцінки знань, що є основною темою статті. У межах автоматизованого оцінювання є проблема оцінювання відкритих текстових відповідей. Викладачі хотіли б знати якість знань учнів, ставлячи відповідні запитання, що вимагають опису процесів мислення. Однак немає жодних автоматизованих рішень, які могли хоча б попередньо оцінити такі відповіді. Рішення, певно, лежить у галузі NLP та видобутку даних. Після деяких досліджень із цих тем автор натрапив на статті, що описують підходи до аналізу даних та тексту, а основа для цих рішень лежить у базах знань. Також у тексті виокремлюються фрагменти інтенсивної логіки з точки зору впровадження технічної системи, яка спрямована на роботу з природними текстовими даними. Розглянуто принципи застосування інтенсивної логіки Р. Монтегю для формалізації текстових даних, представлених природною мовою, з метою створення баз даних знань, необхідних для подальшої реалізації автоматичної генерації навчальних тестів. Далі пропонується практичний підхід до завдання генерації навчальних тестів, заснований на технології вилучення знань із природно-мовних текстів із використанням програмного лінгвістичного процесора.

Abstract PR-12: Towards verifying results from biomedical deep learning models using the UMLS: Cases of primary tumor site classification and cancer Named Entity Recognition

Abstract Background Advances in Machine learning (ML) for biomedical research have led to ground-breaking results that are evident in several healthcare settings. In the field of NLP, there are applications for text mining, named entity recognition, classification of pathology and radiology reports, and text generation. Despite these advancements, the main barrier towards the use of AI systems in a clinical setting is their lack of explainability and interpretability. Objective From the various avenues available through which to create transparent and explainable ML models, we investigated how a stable, accurate, and trusted biomedical standard, the Unified Medical Language System (UMLS) can be applied to retrospectively justify and explain the results of ML models. Methods We developed a novel architecture that places a UMLS-based system after the ML model, and this then acts as a verifier to confirm the accuracy, or lack thereof, of the results from an ML model, and goes on further to explain the results from the model. This architecture is intended to be model-agnostic, so we evaluated its effectiveness using two NLP tasks: classification and Named Entity Recognition (NER). For classification, the UMLS-based verifier was applied to the results from classifying the topographies in 1964 unstructured and anonymized breast cancer pathology reports by a Multi-Task Convolutional Neural Network (MT-CNN). For NER, the UMLS-based verifier was applied to the results of the HunFlair model on unstructured and anonymized breast, colon, and small intestine cancer pathology reports. Results For the classification evaluation, we found that an entity's National Cancer Institute term (NCIt) code can be used to obtain a topographical range for individual entities in a pathology report. We found, further, that, whilst there are entities whose topographical range contributes positively towards a report's overall topography classification, there are also entities that contribute negatively, and that the number of these negative-contributing entities is inversely proportional to the confidence value from the ML model. For the NER evaluation, we found that the UMLS-based verifier is able to both confirm accurate model annotations, and group together the different kinds of inaccuracies found. Additionally, the grouping of an incorrectly tagged entity was found to be correlated with lower confidence values from the model. Conclusion The architecture we propose retrospectively verifies and explains the results of ML models, thus providing a level of interpretability to a model’s outputs. Our use of an industry standard healthcare knowledge repository, the UMLS, is an important contribution towards trusting the results of AI systems in healthcare. Citation Format: Joan Byamugisha, Waheeda Saib, Theodore Gaelejwe, Asad Jeewa, Maletsabisa Molapo. Towards verifying results from biomedical deep learning models using the UMLS: Cases of primary tumor site classification and cancer Named Entity Recognition [abstract]. In: Proceedings of the AACR Virtual Special Conference on Artificial Intelligence, Diagnosis, and Imaging; 2021 Jan 13-14. Philadelphia (PA): AACR; Clin Cancer Res 2021;27(5_Suppl):Abstract nr PR-12.

Semantic Matching for Short Texts: A Cross Attention Mechanism

Semantic matching is a hot topic in the field of NLP. It is widely used in information matching, machine translation, intelligent Q&A, and answering system. With the rapid development of artificial intelligence, people pay more attention to friendly human-computer interaction and how to obtain accurate and effective information. Aiming at the problem of the diversity and structure of Chinese, this paper introduces the cross attention mechanism. Texts provide attention to each other, then to information retrieval. Experiments on aetc-nlp dataset show that the method is effective.

Sentiment analysis of commodity reviews based on ALBERT-LSTM

Sentiment analysis of product reviews has now become one of the important research directions in the field of NLP, which is of great significance in helping merchants better understand user preferences and providing decision-making basis for other users to purchase products. Existing product review sentiment analysis models based on deep learning mostly use traditional word vector models. It is difficult to obtain the contextual semantic information of words, and there is a problem of “polysemous one word”. A product review sentiment analysis model combining ALBERT and LSTM is proposed. First, we use the ALBERT pre-training model to get the word vector containing sequence and semantic information. Then we use the LSTM model that can obtain long-distance semantic features for training. Finally, the emotional polarity of product reviews is classified and output through the emotional polarity discrimination layer. The experimental results on the data set of digital products of JD Mall show that the F1 value of the ALBERT-LSTM model is improved compared with the ALBERT and LSTM models.

Towards automatically generating block comments for code snippets

Code commenting is a common programming practice of practical importance to help developers review and comprehend source code. There are two main types of code comments for a method: header comments that summarize the method functionality located before a method, and block comments that describe the functionality of the code snippets within a method. Inspired by the effectiveness of deep learning techniques in the NLP field, many studies focus on using the machine translation model to automatically generate comment for the source code. Because the data set of block comments is difficult to collect, current studies focus more on the automatic generation of header comments than that of block comments. However, block comments are important for program comprehension due to their explanation role for the code snippets in a method. To fill the gap, we have proposed an approach that combines heuristic rules and learning-based method to collect a large number of comment-code pairs from 1,032 open source projects in our previous study. In this paper, we propose a reinforcement learning-based method, RL-BlockCom , to automatically generate block comments for code snippets based on the collected comment-code pairs. Specifically, we utilize the abstract syntax tree (i.e., AST) of a code snippet to generate a token sequence with a statement-based traversal way. Then we propose a composite learning model, which combines the actor-critic algorithm of reinforcement learning with the encoder-decoder algorithm, to generate block comments. On the data set of the comment-code pairs, the BLEU-4 score of our method is 24.28, which outperforms the baselines and state-of-the-art in comment generation.

A Literature Review on Cross Domain Sentiment Analysis Using Machine learning

Sentiment analysis is the field of NLP which analyzes the sentiments of text written by users on online sites in the form of reviews. These reviews may be either in the form of a word, sentence, document, or ratings. These reviews are used as datasets when applied to train a classifier. These datasets are applied in the annotated form with the positive, negative or neutral labels as an input to train the classifier. This trained classifier is used to test other reviews, either in the same or different domains to know like or dislike of the user for the related field. Various researches have been done in single and cross domain sentiment analysis. The new methods proposed are overcoming the previous ones but according to this survey, no methods best suit the proposed work. In this article, the authors review the methods and techniques that are given by various researchers in cross domain sentiment analysis and how those are compared with the pre-existing methods for the related work.

Towards optimize-ESA for text semantic similarity: A case study of biomedical text

Explicit Semantic Analysis (ESA) is an approach to measure the semantic relatedness between terms or documents based on similarities to documents of a references corpus usually Wikipedia. ESA usage has received tremendous attention in the field of natural language processing NLP and information retrieval. However, ESA utilizes a huge Wikipedia index matrix in its interpretation by multiplying a large matrix by a term vector to produce a high-dimensional vector. Consequently, the ESA process is too expensive in interpretation and similarity steps. Therefore, the efficiency of ESA will slow down because we lose a lot of time in unnecessary operations. This paper propose enhancements to ESA called optimize-ESA that reduce the dimension at the interpretation stage by computing the semantic similarity in a specific domain. The experimental results show clearly that our method correlates much better with human judgement than the full version ESA approach.

A method of automatic text summarisation based on long short-term memory

Deep learning is currently developing very fast in the NLP field and has achieved many amazing results in the past few years. Automatic text summarisation means that the abstract of the document is automatically summarised by a computer program without changing the original intention of the document. There are many application scenarios for automatic summarisation, such as news headline generation, scientific document abstract generation, search result segment generation, and product review summarisation. In the era of internet big data in the information explosion, if the short text can be employed to express the main connotation of information, it will undoubtedly help to alleviate the problem of information overload. In this paper, a model based on the long short-term memory network is presented to automatically analyse and summarise Chinese articles by using the seq2seq+attention models. Finally, the experimental results are attached and evaluated.

The Lao Text Classification Method Based on KNN

Abstract Text categorization is a common application scenario in the NLP field, and has many applications in public opinion monitoring and news classification. At present, there are few classifications for Lao text, but classification-oriented methods are widely used in other languages. Faced with the situation, this paper proposes a KNN-based classification method of Lao news text. First of all, preprocessing and feature extraction of Lao news text, then adjusting the parameters through KNN classifier, finally processing in data normalization and data dimensionality reduction, thus improving the classification effect.

Named entity recognition in steel field based on BiLSTM-CRF model

NER is a basic research in the field of NLP. Most of the Chinese NER tasks are for the recognition of names of people, places, and institutions. The recognized content is single and cannot be applied in many fields. In this paper, we label and build the NER dataset for the steel field. Based on the BiLSTM-CRF model, we proposed character and word combination embedding method. Experiments show that our method reduces the OOV problem, which has an increase in 7.20% and 5.03% in the F1 score compared with the traditional character embedding and word embedding.

Towards Identifying Author Confidence in Biomedical Articles

In an era where the volume of medical literature is increasing daily, researchers in the biomedical and clinical areas have joined efforts with language engineers to analyze the large amount of biomedical and molecular biology literature (such as PubMed), patient data, or health records. With such a huge amount of reports, evaluating their impact has long stopped being a trivial task. In this context, this paper intended to introduce a non-scientific factor that represents an important element in gaining acceptance of claims. We postulated that the confidence that an author has in expressing their work plays an important role in shaping the first impression that influences the reader’s perception of the paper. The results discussed in this paper were based on a series of experiments that were ran using data from the open archives initiative (OAI) corpus, which provides interoperability standards to facilitate effective dissemination of the content. This method may be useful to the direct beneficiaries (i.e., authors, who are engaged in medical or academic research), but also, to the researchers in the fields of biomedical text mining (BioNLP) and NLP, etc.

Idiomaticity Prediction of Chinese Noun Compounds and Its Applications

Idiomaticity refers to the situation where the meaning of a lexical unit cannot be derived from the usual meanings of its constituents. As a ubiquitous phenomenon in languages, the existence of idioms often causes significant challenges for semantic NLP tasks. While previous research mostly focuses on the idiomatic usage detection of English verb-noun combinations and the semantic analysis of Noun Compounds (NCs), the idiomaticity issues of Chinese NCs have been rarely studied. In this work, we aim at classifying Chinese NCs into four idiomaticity degrees. Each idiomaticity degree refers to a specific paradigm of how the NCs should be interpreted. To address this task, a Relational and Compositional Representation Learning model (RCRL) is proposed, which considers the relational textual patterns and the compositionality levels of Chinese NCs. RCRL learns relational representations of NCs to capture the semantic relations between two nouns within an NC, expressed by textual patterns and their statistical signals in the corpus. It further employs compositional representations to model the compositionality levels of NCs via network embeddings. Both loss functions of idiomaticity degree classification and representation learning are jointly optimized in an integrated neural network. Experiments over two datasets illustrate the effectiveness of RCRL, outperforming state-of-the-art approaches. Three applicational studies are further conducted to show the usefulness of RCRL and the roles of idiomaticity prediction of Chinese NCs in the fields of NLP.

Deep Learning Models For Word Sense Disambiguation: A Comparative Study

A specific word can have different meanings depending on the context in which it appears. Identifying the proper sense of the word is crucial in many tasks such as Machine Translation, Anaphora Resolution, Search Engine Recommendation. A word with wrong sense can make the whole sentence meaningless. The task of Word Sense Disambiguation (WSD), is to assign a sense to the ambiguous word based on the context. This is a basic problem which needs to be resolved in the field of NLP and has a variety of solutions. Usual approaches involve supervised machine learning techniques which uses bag-of-words approach. But to achieve better results, WSD should move to sequence modeling rather than the traditional bag of words approach for better performance. Deep neural networks have been used for variety of NLP tasks and a Bi-Directional Long Short-Term Memory (BiLSTM) or a Bi-Directional Gated Recurrent Unit (BiGRU) are possible candidate solutions. The proposed model uses one word per BiGRU or BiLSTM approach, where each word has a model trained for disambiguation. This helps in easy modification of existing model whenever needed. A comparative study on various deep learning models is also performed. The evaluation study of the models shows Bi-Directional models outperforming the other deep learning models.