Language Processing In Recent Years Research Articles

Sentiment analysis of COP9-related tweets: a comparative study of pre-trained models and traditional techniques.

Sentiment analysis has become a crucial area of research in natural language processing in recent years. The study aims to compare the performance of various sentiment analysis techniques, including lexicon-based, machine learning, Bi-LSTM, BERT, and GPT-3 approaches, using two commonly used datasets, IMDB reviews and Sentiment140. The objective is to identify the best-performing technique for an exemplar dataset, tweets associated with the WHO Framework Convention on Tobacco Control Ninth Conference of the Parties in 2021 (COP9). A two-stage evaluation was conducted. In the first stage, various techniques were compared on standard sentiment analysis datasets using standard evaluation metrics such as accuracy, F1-score, and precision. In the second stage, the best-performing techniques from the first stage were applied to partially annotated COP9 conference-related tweets. In the first stage, BERT achieved the highest F1-scores (0.9380 for IMDB and 0.8114 for Sentiment 140), followed by GPT-3 (0.9119 and 0.7913) and Bi-LSTM (0.8971 and 0.7778). In the second stage, GPT-3 performed the best for sentiment analysis on partially annotated COP9 conference-related tweets, with an F1-score of 0.8812. The study demonstrates the effectiveness of pre-trained models like BERT and GPT-3 for sentiment analysis tasks, outperforming traditional techniques on standard datasets. Moreover, the better performance of GPT-3 on the partially annotated COP9 tweets highlights its ability to generalize well to domain-specific data with limited annotations. This provides researchers and practitioners with a viable option of using pre-trained models for sentiment analysis in scenarios with limited or no annotated data across different domains.

Aspect-Based Sentiment Analysis of Specific Targets

Different from traditional sentiment analysis tasks, Aspect-Based Sentiment Analysis (ABSA) aims to automatically identify the emotional polarity of different aspects in the same sentence, which helps to mine users' more delicate emotional expressions for different targets and has become a research hotspot in the field of natural language processing in recent years. Thanks to the rapid development of attention-based deep neural network models, the accuracy of aspect sentiment analysis has continuously made breakthroughs. However, existing works pay little attention to the performance bounds of different application scenarios, such as different text topics or text lengths. In response to the above issues, this paper conducts a comparative analysis of several classic end-to-end neural network models to explore topic-level sentiment polarity analysis. By quantitatively evaluating the accuracy and F1 score of each model at different fixed text lengths, this paper summarizes the strengths, weaknesses, and different aspects of each model's efficiency in use. In addition, this paper also discusses the existing problems with these models and puts forward suggestions for improvement, which can provide some new insights into the research of ABSA.

Accounting fraud detection using contextual language learning

Accounting fraud is a widespread problem that causes significant damage in the economic market. Detection and investigation of fraudulent firms require a large amount of time, money, and effort for corporate monitors and regulators. In this study, we explore how textual contents from financial reports help in detecting accounting fraud. Pre-trained contextual language learning models, such as BERT, have significantly advanced natural language processing in recent years. We fine-tune the BERT model on Management Discussion and Analysis (MD&A) sections of annual 10-K reports from the Securities and Exchange Commission (SEC) database. Our final model outperforms the textual benchmark model and the quantitative benchmark model from the previous literature by 15% and 12%, respectively. Further, our model identifies five times more fraudulent firm-year observations than the textual benchmark by investigating the same number of firms, and three times more than the quantitative benchmark. Optimizing this investigation process, where more fraudulent observations are detected in the same size of the investigation sample, would be of great economic significance for regulators, investors, financial analysts, and auditors.

GPT classifications, with application to credit lending

Generative Pre-trained Transformers (GPT) and Large language models (LLMs) have made significant advancements in natural language processing in recent years. The practical applications of LLMs are undeniable, rendering moot any debate about their impending influence. The power of LLMs has made them similar to machine learning models for decision-making problems. In this paper, we focus on binary classification which is a common use of ML models, particularly in credit lending applications. We show how a GPT model can perform almost as accurately as a classical logistic machine learning model but with a much lower number of sample observations. In particular, we show how, in the context of credit lending, LLMs can be improved and reach performances similar to classical logistic regression models using only a small set of examples.

The Neuroscience of Music Processing and Music Interventions for Mental Illness

Growing research on music and language processing in recent years has simultaneously enabled us to understand its uses and promise in therapeutic settings. This review used recently published peer-reviewed literature to study the neurobiology of music processing, which includes the auditory processing hierarchy and engagement with language processes. Results showed that music has an impact on emotion systems, particularly the reward pathway and the Hypothalamic-Pituitary-Adrenal (HPA) axis. Furthermore, listening to music or music therapy for the treatment of psychiatric and neurological diseases demonstrated effectiveness in improving patient outcomes. Overall, music processing is a sophisticated process with some similarities to language processing. Future randomized controlled trials are required to shed light on the neurocognitive aspect of music processing.

Enhancing local citation recommendation with recurrent highway networks and SciBERT-based embedding

When writing academic papers, referencing statements, claims and previous studies is always an important activity. However, it is considered challenging for scientists to find relevant and appropriate scientific articles which are closely related to their current works in order to reference them in their research. As a consequence of the rapid growth in scientific papers being published every year, researchers might easily get overwhelmed within a huge number of resources. One way to help them find the desired references more easily is to use context-aware citation recommendation. The citation recommender system can automatically provide a list of suitable papers as references based on specified inputs which reflect the researchers’ interests. Among the outstanding achievements of deep learning and natural language processing in recent years, the utilization of deep neural learning architectures have supported to address the problem of citation recommendation. As the result, the neural citation recommendation area has received much attention from the academic community, with the aim of enhancing the precision and correctness of the results of existing citation recommendation systems. Following this research direction, in our paper we present a novel context-aware citation recommendation model, called RHN-DualLCR (Recurrent Highway Networks – Dual Local Citation Recommendation), which integrates Recurrent Highway Networks (RHN), an improved model of the original Bidirectional Long Short-Term Memory (BiLSTM) architecture, and uses a SciBERT-based (Science text of Bidirectional Encoder Representations from Transformers) embedding layer to build up the efficiency of the state-of-the-art local citation recommendation model, which enriches context representation with global information. Our research demonstrates its originality and relevance because we used one of the latest achievements of deep models (RHN model) and natural language processing (SciBERT) applied to the citation recommendation problem. We have conducted experiments on the RHN-DualLCR model on 3 widely known datasets for the citation recommendation problem: ACL-200 (Association for Computational Linguistics), ACL-600 and RefSeer, and also used 2 common evaluation standards Mean Reciprocal Rank (MRR) and the Recall@K (R@K for short) to evaluate the performance of our model. Experimental results show that our proposed model is 3% to 16% better than the original models or state-of-the-art models.

Using Fully Convolutional Network to Locate Transcription Factor Binding Sites Based on DNA Sequence and Conservation Information.

Transcription factors (TFs) play a part in gene expression. TFs can form complex gene expression regulation system by combining with DNA. Thereby, identifying the binding regions has become an indispensable step for understanding the regulatory mechanism of gene expression. Due to the great achievements of applying deep learning (DL) to computer vision and language processing in recent years, many scholars are inspired to use these methods to predict TF binding sites (TFBSs), achieving extraordinary results. However, these methods mainly focus on whether DNA sequences include TFBSs. In this paper, we propose a fully convolutional network (FCN) coupled with refinement residual block (RRB) and global average pooling layer (GAPL), namely FCNARRB. Our model could classify binding sequences at nucleotide level by outputting dense label for input data. Experimental results on human ChIP-seq datasets show that the RRB and GAPL structures are very useful for improving model performance. Adding GAPL improves the performance by 9.32% and 7.61% in terms of IoU (Intersection of Union) and PRAUC (Area Under Curve of Precision and Recall), and adding RRB improves the performance by 7.40% and 4.64%, respectively. In addition, we find that conservation information can help locate TFBSs.

A novel unsupervised ensemble framework using concept-based linguistic methods and machine learning for twitter sentiment analysis

• A Novel ensemble/cooperative framework based on concept-based and clustering is proposed to perform Twitter sentiment Analysis. • It employs majority voting, tie breaker criteria, and linguistic rules in concept-based module. • Comparative analysis between clustering and classification is presented when integrated with concept based methods. • It presents the performance of feature representation methods (Boolean and TF-IDF). • Experimental results on Twitter Datasets revealed better performance of proposed framework. Concept-based sentiment analysis (CBSA) methods have gained prominence in natural language processing in recent years. These methods consider the underlying semantic meanings of text to perform different tasks such as Twitter sentiment analysis (assigning positive, negative, or neutral sentiment to Tweets). CBSA is superior to traditional statistical methods for accurately discovering sentiment labels. Due to a limited knowledge base, these methods are unable to identify the sentiment polarity of all kinds of text. Therefore, supervised learning techniques are mostly ensembled with CBSA methods to classify the whole text. These techniques require labeled data. It is a tedious and time-consuming task due to the manually labeling of large datasets (Such as Twitter datasets). Therefore, an unsupervised learning mechanism can be a better alternative to solve this problem. In this paper, a novel unsupervised learning framework based on Concept-based and hierarchical clustering is proposed for Twitter sentiment analysis. Popular hierarchical clustering methods including single linkage, complete linkage, and average linkage algorithms are ensembled serially. Two different feature representation methods including Boolean and Term frequency-inverse document frequency (TF-IDF) are investigated. We have also experimented with Well-known classifiers (Naive Bayes, Neural Network) for a fair comparison. Accuracy measure (proportion of correct predictions) is used to evaluate the performance of understudied techniques. It is empirically shown that the performance of unsupervised learning techniques is comparable with supervised learning techniques.

Full-Circuit Implementation of Transformer Network Based on Memristor

As an emerging in-memory element, memristor has been widely used in various neural network circuits to represent the weights and accelerate the calculation. However, the Transformer Network (TN), one of the most important models for machine vision and natural language processing in recent years, has not yet been full-circuit implemented using memristors due to the complex calculation process and data storage. In order to carry out the computation of the TN more efficiently, this work proposes a memristor-based full-circuit implementation of the TN capable of: 1) a memristor crossbar module to preserve the weights of the TN and perform the vector-matrix multiplications; 2) an analog signal memory module to store the analog signal directly in near-memory mode; 3) function circuit modules to achieve five transformations, namely Softmax, Layer Normalization, ReLU, Multiply-add and Residual; 4) a timing signal generation module to schedule operations of the circuit. The proposed TN circuit can complete all calculations directly based on the analog signal without using any analog-digital converter (ADC), digital-analog converter (DAC) and digital memory. In addition, character image recognition experiments are carried out in PSPICE to verify the functional correctness of the designed circuit. The corresponding signal retention rates of the analog memory, the performance of the whole circuit, and the non-idealities of the memristors are also analyzed. The results indicate that the circuit has advantages in terms of area overhead, energy efficiency and anti-noise.

A Review of the Challenges of Using Deep Learning Algorithms to Support Decision-Making in Agricultural Activities

Deep Learning has been successfully applied to image recognition, speech recognition, and natural language processing in recent years. Therefore, there has been an incentive to apply it in other fields as well. The field of agriculture is one of the most important fields in which the application of deep learning still needs to be explored, as it has a direct impact on human well-being. In particular, there is a need to explore how deep learning models can be used as a tool for optimal planting, land use, yield improvement, production/disease/pest control, and other activities. The vast amount of data received from sensors in smart farms makes it possible to use deep learning as a model for decision-making in this field. In agriculture, no two environments are exactly alike, which makes testing, validating, and successfully implementing such technologies much more complex than in most other industries. This paper reviews some recent scientific developments in the field of deep learning that have been applied to agriculture, and highlights some challenges and potential solutions using deep learning algorithms in agriculture. The results in this paper indicate that by employing new methods from deep learning, higher performance in terms of accuracy and lower inference time can be achieved, and the models can be made useful in real-world applications. Finally, some opportunities for future research in this area are suggested.

Data-Driven Microseismic Event Localization: An Application to the Oklahoma Arkoma Basin Hydraulic Fracturing Data

The microseismic monitoring technique is widely applied to petroleum reservoirs to understand the process of hydraulic fracturing. Geophones continuously record the microseismic events triggered by fluid injection on the Earth’s surface or in monitoring wells. The microseismic event localization precision has a large impact on the performance of the technique. Deep learning has achieved significant progress in computer vision and natural language processing in recent years. We propose to use a deep convolutional neural network (CNN) to directly map the field records to their event locations. The biggest advantage of deep learning methods over conventional methods is that they can efficiently predict the characteristics of a huge amount of recorded data without human intervention. Thus, we use a CNN to predict the event location of field microseismic data that were recorded during a hydraulic fracturing process of a shale gas play in Oklahoma, the United States. We use synthetic data with extracted field noise from the records to train CNN. The synthetic training data allow us to produce the corresponding labels, and the extracted noise from the field data reduces the difference between the field and synthetic data. We use a correlation preprocessing step to avoid the need for event detection and picking of arrivals. We demonstrate that the proposed approach provides accurate microseismic event locations at a much faster speed than traditional imaging methods, such as time-reversal imaging. Comparison with an existing study on the same data is presented to evaluate the performance of the trained neural network.

Recent Advancement of Deep Learning Applications to Machine Condition Monitoring Part 1: A Critical Review

With the huge success of applying deep learning (DL) methodologies to image recognition and natural language processing in recent years, researchers are now keen to use them in the machine condition monitoring (MCM) context. There are numerous papers in applying various DL techniques, such as auto-encoder, restricted Boltzmann machine, convolutional neural network and recurrent neural network, etc., to MCM problems ranging from component-level condition monitoring (machine tool wear prediction, bearing fault diagnosis and classification and hydraulic pump fault diagnosis) to system-level health management (aircraft and spacecraft diagnosis). In this paper, we give a brief overview in the area of DL for MCM with a focus on reviewing the most recent papers published since 2019. In Part 1, we present some critical views regarding whether any breakthrough has been achieved from an MCM domain expert perspective, with the main conclusion that DL has great potential for MCM applications, and a major breakthrough could come soon since the shortfalls lie more in data than in the DL methodologies. Our overall impression is that (a) DL models are not really showing their great potentials with only a small training data; (b) faulty-condition data is hard to come by for training DL, but normal condition data is abundant, so anomaly detection makes more sense; (c) applying DL only to the Case Western Reserve University (CWRU) bearing fault dataset is not sufficient for real world industrial applications as it was from a very simple test rig, and applying DL to data from complex systems like helicopter gearbox data may deliver much more convincing results. In Part 2, we enhance the main conclusion of the critical review with supplement views and a case study on analysing Bell-206B helicopter main gearbox planet bearing failure data using some traditional MCM techniques in contrast to applying the long short-term memory (LSTM) DL method. We can conclude from the case study that the DL-based methods are not necessarily always superior to the traditional MCM techniques for dataset from moderately complex machinery.

Research Trends in the International Literature on Natural Language Processing, 2000-2019 — A Bibliometric Study

This work presents a review, using bibliometric methods, of the state of research on the whole field of natural language processing (NLP), understanding this as the methods to process human language, including semantic techniques, statistical techniques or a combination of both. Particularly we focus on the trends of research in NLP, since there are not in the literature studies that embrace in an integral way bibliometric studies about natural language processing, its applications and related topics. Our work includes an identification of the main sources where research is published, the most productive and influential countries and research institutions, the main actors involved in research, as well as the main topics that are investigated. We found that research in the field and subfields has increased continuously during the period under study; conference proceedings are the preferred media to communicate results and that biomedical informatics is one relevant field of application of NLP. We conclude with both, a synchronic and a diachronic characterization of research topics carried out internationally on natural language processing and related topics, which showed that several subfields of artificial intelligence are closely related to natural language processing in recent years.

A Sentiment Analysis Method Based on BLSTM and CNN Fusion

Sentiment analysis is a research hotspot in natural language processing in recent years. This paper proposes a sentiment analysis method that integrates LSTM and CNN in view of the fact that most existing sentiment analysis methods mix semantics and emotions. This method divides the text into semantic space and emotional space, and uses the attention model to train in two dimensions, and fuses the feature representations of the two to construct the final representation of the text. The results show that the model has achieved good results in multiple emotional review data.

The impact of corpus domain on word representation: a study on Persian word embeddings

Word embedding, has been a great success story for natural language processing in recent years. The main purpose of this approach is providing a vector representation of words based on neural network language modeling. Using a large training corpus, the model most learns from co-occurrences of words, namely Skip-gram model, and capture semantic features of words. Moreover, adding the recently introduced character embedding model to the objective function, the model can also focus on morphological features of words. In this paper, we study the impact of training corpus on the results of word embedding and show how the genre of training data affects the type of information captured by word embedding models. We perform our experiments on the Persian language. In line of our experiments, providing two well-known evaluation datasets for Persian, namely Google semantic/syntactic analogy and Wordsim353, is also part of the contribution of this paper. The experiments include computation of word embedding from various public Persian corpora with different genres and sizes while considering comprehensive lexical and semantic comparison between them. We identify words whose usages differ between these datasets resulted totally different vector representation which ends to significant impact on different domains in which the results vary up to 9% on Google analogy and up to 6% on Wordsim353. The resulted word embedding for each of the individual corpora as well as their combinations will be publicly available for any further research based on word embedding for Persian.

Improved New Word Detection Method Used in Tourism Field

Abstract Chinese segmentation has attracted amounts of attention in natural language processing in recent years and is the basis of web text mining. The article improved statistics-based method EMI, then we use improved approach to detect new words in tourism field. The result demonstrates that our method can detect new words significantly, especially in detecting proper nouns and sentiment words which will be helpful in subsequent tasks such as sentiment analysis and word embedding. In additional, this paper analyze parameters which are influential on the effects of new words detection. At last, the article discussed possible application of new word detection in sentiment analysis.

Using relation selection to improve value propagation in a ConceptNet-based sentiment dictionary

Sentiment analysis has become an important topic in natural language processing in recent years, and sentiment dictionaries are essential for research in this field. Concept-level sentiment dictionaries have broader coverage than word-based dictionaries, but they are still insufficient for real-world applications. In our previous work, we used a commonsense knowledge base (ConceptNet) as the foundation to build a larger dictionary. By propagating sentiment values from concepts with known values to empty concepts, we greatly enlarged our concept-level sentiment dictionary. In this work, we refine our previous method by adding relation selection and bias correction steps. Based on the assumption that concepts pass their sentiment values to their neighbors in different ways depending on the relations connecting them, we use sequential forward search to find the best combination of relations. We also propose a bias correction method that guarantees that the average deviation and standard deviation of sentiment values in the whole sentiment dictionary remain unchanged. We show that the strategy can improve the polarity accuracy by 3.7% and the Kendall τ distance by 17.3% relative to our previous method. Also, our experiment shows that the dictionary we constructed leads to better performance in the sentiment polarity classification task.

Joint Optimization for Chinese POS Tagging and Dependency Parsing

Dependency parsing has gained more and more interest in natural language processing in recent years due to its simplicity and general applicability for diverse languages. Previous work demonstrates that part-of-speech (POS) is an indispensable feature in dependency parsing since pure lexical features suffer from serious data sparseness problem. However, due to little morphological changes, Chinese POS tagging has proven to be much more challenging than morphology-richer languages such as English (94% vs. 97% on POS tagging accuracy). This leads to severe error propagation for Chinese dependency parsing. Our experiments show that parsing accuracy drops by about 6% when replacing manual POS tags of the input sentence with automatic ones generated by a state-of-the-art statistical POS tagger. To address this issue, this paper proposes a solution by jointly optimizing POS tagging and dependency parsing in a unique model. We propose for our joint models several dynamic programming based decoding algorithms which can incorporate rich POS tagging and syntactic features. Then we present an effective pruning strategy to reduce the search space of candidate POS tags, leading to significant improvement of parsing speed. Experimental results on two Chinese data sets, i.e. Penn Chinese Treebank 5.1 and Penn Chinese Treebank 7, demonstrate that our joint models significantly improve both the state-of-the-art tagging and parsing accuracies. Detailed analysis shows that the joint method can help resolve syntax-sensitive POS ambiguities <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex Notation="TeX">$\{{\ssr{NN}},{\ssr{VV}}\}$</tex> </formula> . In return, the POS tags become more reliable and helpful for parsing since the syntactic features are used in POS tagging. This is the fundamental reason for the performance improvement.

Dependency Parsing

Dependency-based methods for syntactic parsing have become increasingly popular in natural language processing in recent years. This book gives a thorough introduction to the methods that are most widely used today. After an introduction to dependency grammar and dependency parsing, followed by a formal characterization of the dependency parsing problem, the book surveys the three major classes of parsing models that are in current use: transition-based, graph-based, and grammar-based models. It continues with a chapter on evaluation and one on the comparison of different methods, and it closes with a few words on current trends and future prospects of dependency parsing. The book presupposes a knowledge of basic concepts in linguistics and computer science, as well as some knowledge of parsing methods for constituency-based representations.

動作の時間的連続性を考慮したＢｏｏｓｔｉｎｇアルゴリズムに基づくオンライン日常動作認識

In this paper, we propose a robust online action recognition method based on boosted sequential classification. Our method utilizes boosting algorithm that is one of ensemble learning algorithms. This algorithm is also known as a feautre selector and has been utilized in the fields of image processing and natural language processing in recent years. Based on the boosting scheme, our method can automatically and efficiently select significant features for action recognition. Additionally, the method leverages temporal dependency of actions based on Ising model to improve recognition performance. We evaluated our method to action recognition, such as walking and running, using motion capture data only with posture features. In the result, our method can classify the actions more robustly than the method that does not utilize temporal dependency of actions.