Field Of NLP Research Articles

ЕND-TO-END SPEECH RECOGNITION SYSTEMS FOR AGGLUTINATIVE LANGUAGES

With the improvement of intelligent systems, speech recognition technologies are being widely integrated into various aspects of human life. Speech recognition is applied to smart assistants, smart home infrastructure, the call center applications of banks, information system components for impaired people, etc. But these facilities of information systems are available only for common languages, like English, Chinese, or Russian. For low-resource language, these opportunities for information technologies are still not implemented. Most modern speech recognition approaches are still not tested on agglutinative languages, especially for the languages of Turkic group like Kazakh, Tatar, and Turkish Languages. The HMM-GMM (Hidden Markov Models - Gaussian Mixture Models) model has been the most popular in the field of Automatic Speech Recognition (ASR) for a long time. Currently, neural networks are widely used in different fields of NLP, especially in automatic speech recognition. In an enormous number of works application of neural networks within different stages of automatic speech recognition makes the quality level of this systems much better. Integral speech recognition systems based on neural networks are investigated in the article. The paper proves that the Connectionist Temporal Classification (CTC) model works precisely for agglutinative languages. The author conducted an experiment with the LSHTM neural network using an encoder-decoder model, which is based on the attention-based models. The result of the experiment showed a Character Error Rate (CER) equal to 8.01% and a Word Error Rate (WER) equal to 17.91%. This result proves the possibility of getting a good ASR model without the use of the Language Model (LM).

Punctuation Prediction in Bangla Text

Punctuation prediction is critical as it can enhance the readability of machine-transcribed speeches or texts significantly by adding appropriate punctuation. Furthermore, systems like Automatic Speech Recognizer (ASR) produce texts that are unpunctuated, making the readability difficult for humans and also hampers the performance of various natural language processing (NLP) tasks. Such NLP related tasks have been investigated thoroughly for English; however, very limited work is done for punctuation prediction in the Bangla language. In this study, we train a bidirectional recurrent neural network (BRNN) along with Attention model with a plausibly large Bangla dataset. Afterwards, we apply extensive postprocessing techniques for predicting punctuation more accurately with the employed model. Initially, we perform experimentation with a relatively imbalanced dataset, and our model shows promising results F1=56.9 for Period) in punctuation prediction. Later, we also investigate the model’s performance using a balanced Bangla dataset to achieve higher performance scores ( F1=62.2 for Question). Thus, the goal of this study is to propose an efficient approach that can predict punctuation in Bangla texts effectively. Our study also includes investigation on how our postprocessing techniques affect the prediction performance. Being an early attempt for the punctuation prediction in Bangla text, our work is expected to significantly contribute in the NLP field for the Bangla language, and will pave the way for future work with the Bangla language in this direction.

Conversational AI: Dialogue Systems, Conversational Agents, and Chatbots by Michael McTear

Conversational AI: Dialogue Systems, Conversational Agents, and Chatbots by Michael McTear

Metaphor Recognition Method based on Graph Neural Network

This Metaphor is a very common language phenomenon. Human language often uses metaphor to express emotion, and metaphor recognition is also an important research content in the field of NLP. Official documents are a serious style and do not usually use rhetorical sentences. This paper aims to identify rhetorical metaphorical sentences in official documents. The use of metaphors in metaphorical sentences depends on the context. Based on this linguistic feature, this paper proposes a BertGAT model, which uses Bert to extract semantic features of sentences and transform the dependency relationship between Chinese text and sentences into connected graphs. Finally, the graph attention neural network is used to learn semantic features and syntactic structure information to complete sentence metaphor recognition. The proposed model is tested on the constructed domain dataset and the sentiment public dataset respectively. Experimental results show that the method proposed in this paper can effectively improve the recognition ability of metaphorical emotional sentences.

Methods of eliminating homonymy within different, grammatically similar word groups

The problem of automatic processing of natural language remains relevant for more than half a century. One of the important problems in the field of NLP is the creation of a semantic analyzer, which in turn goes through a number of steps. Determining homonymy is important in the semantic analysis of sentences. A method based on rules, a method based on statistical data, and methods based on machine learning are also used to determine homonymy. Statistical methods are mainly used to determine homonymy between grammatically similar word groups. In this article discusses the use of homonymy between two grammatically similar nouns and adjectives using statistical methods, namely Frequency and Bayesian methods. If bigrams and trigrams are used in the Bayesian method, the characteristics of word groups are classified in the frequentist method, and the parameters that can distinguish them are determined. The identified parameters are converted into numbers as a result of observations and probabilistic decisions are made.

DESIGN AND IMPLEMENTATION OF A RULE BASED AFAAN OROMOO CONVERSATIONAL CHATBOTS

Chatbot technologies are very popular one in the field of NLP and AI to process basic expressions of interaction between humans (end users) and devices as Question Answering System (QAS). They are used in many ranges like in industry to provide customer services based on their need to reach on their desires, transportation, health, entertainment, education, psychology to monitor human emotions and in many. Messaging service is a popular means of communication as text based messaging (SMS) or through messaging applications. Chatbots are virtual assistant technologies that are designed and developed based on the user’s query and to their language. The aim of this paper is to develop and implement a rule based conversational chatbot for Afaan Oromo Language. The approach gets a simple Afaan Oromoo query as input and retrieves answers from a set of predefined rules or responses. The performance of our model is evaluated through human judgment.

A method of named entity recognition for Tigrinya

This paper proposes a method for Named-Entity Recognition (NER) for a low-resource language, Tigrinya, using a pre-trained language model. Tigrinya is a morphologically rich, although one of the underrepresented in the field of NLP. This is mainly due to the limited amount of annotated data available. To address this problem, we present the first publicly available datasets of NER for Tigrinya containing two versions, namely, (V1 and V2) annotated manually. The V1 and V2 datasets contain 69,309 and 40,627 tokens, respectively, where the annotations are based on the CoNLL 2003 Beginning, Inside, and Outside (BIO) tagging schema. Specifically, we develop a new pre-trained language model for Tigrinya based on RoBERTa, which we refer to as TigRoBERTa. Our model is then fine-tuned on down-stream tasks on a more specific target NER and POS tasks with limited data. Finally, we further enhance the model performance by applying semi-supervised self-training using unlabeled data. The experimental results show that the method achieved 84% F1-score for NER and 92% accuracy for POS tagging, which is better than or comparable to the baseline method based on the CNN-BiLSTM-CRF.

Accurate prediction of virus-host protein-protein interactions via a Siamese neural network using deep protein sequence embeddings

SummaryPrediction and understanding of virus-host protein-protein interactions (PPIs) have relevance for the development of novel therapeutic interventions. In addition, virus-like particles open novel opportunities to deliver therapeutics to targeted cell types and tissues. Given our incomplete knowledge of PPIs on the one hand and the cost and time associated with experimental procedures on the other, we here propose a deep learning approach to predict virus-host PPIs. Our method (Siamese Tailored deep sequence Embedding of Proteins [STEP]) is based on recent deep protein sequence embedding techniques, which we integrate into a Siamese neural network. After showing the state-of-the-art performance of STEP on external datasets, we apply it to two use cases, severe acute respiratory syndrome coronavirus 2 and John Cunningham polyomavirus, to predict virus-host PPIs. Altogether our work highlights the potential of deep sequence embedding techniques originating from the field of NLP as well as explainable artificial intelligence methods for the analysis of biological sequences.

Formula omitted]-Scaled-attention: A novel fast attention mechanism for efficient modeling of protein sequences

Attention-based deep networks have been successfully applied on textual data in the field of NLP. However, their application on protein sequences poses unexplored challenges because, unlike the plain text words, protein words have weak semantics. Such challenges include: (i) vanishing attention score problem and (ii) high variations in the attention distribution. Here, we introduce a novel λ-scaled attention technique for a fast and efficient modeling of the protein sequences aimed at addressing these two problems. This is then used to develop the λ-scaled attention network and is evaluated for the task of protein function prediction implemented at the protein sub-sequence level. Experiments on the datasets for biological process (BP) and molecular function (MF) showed significant improvements in the F1 score values for the proposed λ-scaled attention technique over its counterpart approach based on the standard attention technique (+1.68% for BP and +5.27% for MF) and state-of-the-art multi-segment based ProtVecGen-Plus approach (+4.70% for BP and +5.30% for MF). Further, fast convergence (converging in half the number of epochs) and efficient learning (captured by low difference between the training and validation losses) were also observed during the training process.

Natural language processing in toxicology: Delineating adverse outcome pathways and guiding the application of new approach methodologies

Adverse Outcome Pathways (AOPs) are conceptual frameworks that tie an initial perturbation (molecular initiating event) to a phenotypic toxicological manifestation (adverse outcome), through a series of steps (key events). They provide therefore a standardized way to map and organize toxicological mechanistic information. As such, AOPs inform on key events underlying toxicity, thus supporting the development of New Approach Methodologies (NAMs), which aim to reduce the use of animal testing for toxicology purposes.However, the establishment of a novel AOP relies on the gathering of multiple streams of evidence and information, from available literature to knowledge databases. Often, this information is in the form of free text, also called unstructured text, which is not immediately digestible by a computer. This information is thus both tedious and increasingly time-consuming to process manually with the growing volume of data available. The advancement of machine learning provides alternative solutions to this challenge. To extract and organize information from relevant sources, it seems valuable to employ deep learning Natural Language Processing techniques.We review here some of the recent progress in the NLP field, and show how these techniques have already demonstrated value in the biomedical and toxicology areas. We also propose an approach to efficiently and reliably extract and combine relevant toxicological information from text. This data can be used to map underlying mechanisms that lead to toxicological effects and start building quantitative models, in particular AOPs, ultimately allowing animal-free human-based hazard and risk assessment.

NLP4NLP+5: The Deep (R)evolution in Speech and Language Processing.

This paper aims at analyzing the changes in the fields of speech and natural language processing over the recent past 5 years (2016–2020). It is in continuation of a series of two papers that we published in 2019 on the analysis of the NLP4NLP corpus, which contained articles published in 34 major conferences and journals in the field of speech and natural language processing, over a period of 50 years (1965–2015), and analyzed with the methods developed in the field of NLP, hence its name. The extended NLP4NLP+5 corpus now covers 55 years, comprising close to 90,000 documents [+30% compared with NLP4NLP: as many articles have been published in the single year 2020 than over the first 25 years (1965–1989)], 67,000 authors (+40%), 590,000 references (+80%), and approximately 380 million words (+40%). These analyses are conducted globally or comparatively among sources and also with the general scientific literature, with a focus on the past 5 years. It concludes in identifying profound changes in research topics as well as in the emergence of a new generation of authors and the appearance of new publications around artificial intelligence, neural networks, machine learning, and word embedding.

Deep Learning based Handwritten Polynomial Equation Solver

Abstract: Polynomials are algebraic expressions involving a sum of powers in one or more variables multiplied by its coefficients. If x is a variable, a0x n +a1x n-1 +a2x n-2 +....+an then it's a (n) powers polynomial. Human is capable to solve this type of mathematical problems. In this work, we propose a system in which machines can achieve the cognitional skills that can understand the problem by visual context. By taking an input image of Handwritten polynomial equations and simplifies the problem by generating the answer as an output. Here machine can able to solve quadratic, cubic, quartic, quantic, sextic as well as (n) powers polynomials. This proposed work can be workable in an embedded system as well as a mobile application. In this scope for recognition purposes, we use a CNN model. Robust handwritten character recognition is a tricky job in the area of image processing. Among all the problem handwritten mathematical expression recognition is one of the complicated issue in the area of computer vision research. Segmentation and classification of specific character makes the task more difficult. In this paper a group of handwritten quadratic equation as well as a single quadratic equation are considered to recognize and make a solution for those equations. Horizontal compact projection analysis and combined connected component analysis methods are used for segmentation. For classification of specific character we apply Convolutional Neural Network. Each of the correct detection, character string operation is used for the solution of the equation. The proposed workflow system automatically simplifies the Handwritten polynomial equation and has been done a really good performance. Developing an automatic equation recognizer and solver has been a desire of the researchers who worked in the field of NLP for many years.

Slr-kit: A semi-supervised machine learning framework for systematic literature reviews

Systematic Literature Review (SLR) is nowadays a challenging task due to the large number of papers that typically compose the scientific material of the topic to review. Recently, a lot of research effort has been devoted to automate, even partially, the stages of an SLR. This paper proposes the design and implementation of a workflow and a set of tools – called slr-kit – to support key tasks in an SLR. The proposed approach leverages a semi-supervised strategy, in which time-consuming processes are carried out using automatic tools, whereas manual tasks have been optimized by carefully designed support tools to reduce the overall required effort. Important parts of the workflow include the extraction of key terms directly from the abstracts of the papers to survey, and the subsequent topic modeling that allows for a thematic clustering of the corpus of papers. In the proposed workflow, the former task is carried out by exploiting a novel tool, called FAst WOrd Classifier (FAWOC). The latter, instead, is designed to be automatically carried out by leveraging an ad-hoc solution based on the application of the Latent Dirichlet Allocation (LDA) algorithm. The result of the process consists in a set of statistics regarding the relationship among papers, topics, and their trend of publication on journals and conference proceedings. The validity of the method is demonstrated with an application to a dataset related to the scientific field of NLP, while its accuracy is assessed by the manual examination of the results by domain experts.

SwinF: Swin Transformer with feature fusion in target detection

In recent years, transformer has made great achievements in the field of NLP and is gradually applied to Computer Vision. However, due to the particularity of images, the computational complexity of transformer is quite high. The windowing operation proposed by Swin transformer effectively solves this problem. We find that Swin transformer has the same hierarchical structure as CNN, so we propose SwinF network with feature fusion based on Swin transformer. On the coco type dataset, Swin transformer achieves 40.3mAP, while SwinF achieves 42.5mAP in the field of target detection.

Approach to temporal concept localization in videos

Localizing moments in the videos has been a new challenging task in the field of Computer Science to provide faster search time for video retrieval, query processing and also behavioral analysis. The process involves stages such as video understanding, video segmentation, query processing using NLP and generation of localization of the concepts in the video. Though there have been many attempts to Video understanding in the field of NLP and Computer Vision in past years, they lack to cover the large untrimmed videos in current real-life scenarios. We propose the deep learning-based solution with the use of Random Forest and Bi-LSTM approach to localize the labels in segments and also the time at which they pertain to the particular segments. We used the YouTube 8M dataset provided by YouTube in Kaggle's challenge to train our frame-based model and use it to classify the segments using sliding windows of size 5. Our approach tries to provide a naïve and robust approach to model this concept and provide a way to tackle this large problem. Further improvements in the Bi-LSTM based models and Random Forest models with VLAD would lead to better results.

Text Sentiment Analysis Based on Transformer and Augmentation.

With the development of Internet technology, social media platforms have become an indispensable part of people’s lives, and social media have been integrated into people’s life, study, and work. On various forums, such as Taobao and Weibo, a large number of people’s footprints are left all the time. It is these chats, comments, and other remarks with people’s emotional evaluations that make up part of public opinion. Analysis of this network public opinion is conducive to maintaining the peaceful development of society. Therefore, sentiment analysis has become a hot research field and has made great strides as one of the hot topics in the field of natural language processing. Currently, the BERT model and its variants have achieved excellent results in the field of NLP. However, these models cannot be widely used due to huge demands on computing resources. Therefore, this paper proposes a model based on the transformer mechanism, which mainly includes two parts: knowledge distillation and text augmentation. The former is mainly used to reduce the number of parameters of the model, reducing the computational cost and training time of the model, and the latter is mainly used to expand the task text so that the model can achieve excellent results in the few-sample sentiment analysis task. Experiments show that our model achieves competitive results.

A Comparative Analysis on Suicidal Ideation Detection Using NLP, Machine, and Deep Learning

Social networks are essential resources to obtain information about people’s opinions and feelings towards various issues as they share their views with their friends and family. Suicidal ideation detection via online social network analysis has emerged as an essential research topic with significant difficulties in the fields of NLP and psychology in recent years. With the proper exploitation of the information in social media, the complicated early symptoms of suicidal ideations can be discovered and hence, it can save many lives. This study offers a comparative analysis of multiple machine learning and deep learning models to identify suicidal thoughts from the social media platform Twitter. The principal purpose of our research is to achieve better model performance than prior research works to recognize early indications with high accuracy and avoid suicide attempts. We applied text pre-processing and feature extraction approaches such as CountVectorizer and word embedding, and trained several machine learning and deep learning models for such a goal. Experiments were conducted on a dataset of 49,178 instances retrieved from live tweets by 18 suicidal and non-suicidal keywords using Python Tweepy API. Our experimental findings reveal that the RF model can achieve the highest classification score among machine learning algorithms, with an accuracy of 93% and an F1 score of 0.92. However, training the deep learning classifiers with word embedding increases the performance of ML models, where the BiLSTM model reaches an accuracy of 93.6% and a 0.93 F1 score.

Analysis of publications on natural language processing indexed in RSCI

Natural Language Рrocessing (NLP), defined, as the general direction of artificial intelligence and mathematical linguistics, is an important tool for understanding and processing a gigantic amount of unstructured data.Representing theoretical and practical basis for solving a large number of problems, NLP is being actively studied by scientists around the world, since it is one of the most popular areas of data science and is used in many fields of human activity. The purpose of this bibliometric analysis is to identify the main research centers, research trends, the rating of leading Russian scientists in the field of NLP, as well as to show the picture of thematic fields’ distribution of publications. The results of the analysis will allow us to determine the dynamics of NLP field development in the domestic science and will provide scientists and specialists working in the field of NLP with up-to-date information on various aspects of the considered research area.Russian Science Citation Index (RSCI) – the main scientometric database in our country was used as the data source for scientific publications in the field of NLP search. The RSCI platform contains information on current research areas, allows you to assess the productivity of scientists, scientific communities, etc.The presented work will be interesting to researchers in the field of NLP, since it contains actual data on the development dynamics and structure of the documentary and information flow in the field of NLP in Russia, its distribution by branches of knowledge. Besides, the conducted bibliometric analysis of documents allows us to gain information about the authors most productively working in the studied area and their affiliations, as well as about the most cited articles – this is the review of the 30 most cited publications of Russian scientists and rating of TOP-10 scientific organizations of the Russian Federation by the number of publications reflecting the main problems and scientific achievements in the field of NLP.

The Structure of Associative Heterarchical Memory

Natural language processing by artificial intelligence (NLP) remains an urgent problem of our time. The main task of NLP is to create programs capable of processing and understanding natural languages. To solve these problems the authors have proposed a new mathematical formalism - the associative-heterarchical memory (AH-memory), which structure and functioning are based both on bionic principles and on the achievements of both paradigms of artificial intelligence. This article will provide a comprehensive description of AH-memory. The article will be of interest to developers of artificial intelligence and specialists in the field of NLP.

NoiER: An Approach for Training More Reliable Fine-Tuned Downstream Task Models

The recent development in pretrained language models trained in a self-supervised fashion, such as BERT, is driving rapid progress in the field of NLP. However, their brilliant performance is based on leveraging syntactic artifacts of the training data rather than fully understanding the intrinsic meaning of language. The excessive exploitation of spurious artifacts causes a problematic issue: The distribution collapse problem, which is the phenomenon that the model fine-tuned on downstream tasks is unable to distinguish out-of-distribution (OOD) sentences while producing a high confidence score. In this paper, we argue that distribution collapse is a prevalent issue in pretrained language models and propose noise entropy regularisation (NoiER) as an efficient learning paradigm that solves the problem without auxiliary models and additional~data. The proposed approach improved traditional OOD detection evaluation metrics by 55% on average compared to the original fine-tuned models.