Deep Contextualized Word Representations Research Articles

Unmasking deception: a CNN and adaptive PSO approach to detecting fake online reviews.

Online reviews play a critical role in modern word-of-mouth communication, influencing consumers' shopping preferences and purchase decisions, and directly affecting a company's reputation and profitability. However, the credibility and authenticity of these reviews are often questioned due to the prevalence of fake online reviews that can mislead customers and harm e-commerce's credibility. These fake reviews are often difficult to identify and can lead to erroneous conclusions in user feedback analysis. This paper proposes a new approach to detect fake online reviews by combining convolutional neural network (CNN) and adaptive particle swarm optimization with natural language processing techniques. The approach uses datasets from popular online review platforms like Ott, Amazon, Yelp, TripAdvisor, and IMDb and applies feature selection techniques to select the most informative features. The paper suggests using attention mechanisms like bidirectional encoder representations from transformers and generative pre-trained transformer, as well as other techniques like Deep contextualized word representation, word2vec, GloVe, and fast Text, for feature extraction from online review datasets. The proposed method uses a multimodal approach based on a CNN architecture that combines text data to achieve a high accuracy rate of 99.4%. This outperforms traditional machine learning classifiers in terms of accuracy, recall, and F measure. The proposed approach has practical implications for consumers, manufacturers, and sellers in making informed product choices and decision-making processes, helping maintain the credibility of online consumer reviews. The proposed model shows excellent generalization abilities and outperforms conventional discrete and existing neural network benchmark models across multiple datasets. Moreover, it reduces the time complexity for both training and testing.

Context-Aware Sentiment Analysis using Tweet Expansion Method

The large source of information space produced by the plethora of social media platforms in general and microblogging in particular has spawned a slew of new applications and prompted the rise and expansion of sentiment analysis research. We propose a sentiment analysis technique that identifies the main parts to describe tweet intent and also enriches them with relevant words, phrases, or even inferred variables. We followed a state-of-the-art hybrid deep learning model to combine Convolutional Neural Network (CNN) and the Long Short-Term Memory network (LSTM) to classify tweet data based on their polarity. To preserve the latent relationships between tweet terms and their expanded representation, sentence encoding and contextualized word embeddings are utilized. To investigate the performance of tweet embeddings on the sentiment analysis task, we tested several context-free models (Word2Vec, Sentence2Vec, Glove, and FastText), a dynamic embedding model (BERT), deep contextualized word representations (ELMo), and an entity-based model (Wikipedia). The proposed method and results prove that text enrichment improves the accuracy of sentiment polarity classification with a notable percentage.

Spam detection on social networks using deep contextualized word representation

Spam detection on social networks using deep contextualized word representation

Exploring deep learning methods for recognizing rare diseases and their clinical manifestations from texts

Background and objectiveAlthough rare diseases are characterized by low prevalence, approximately 400 million people are affected by a rare disease. The early and accurate diagnosis of these conditions is a major challenge for general practitioners, who do not have enough knowledge to identify them. In addition to this, rare diseases usually show a wide variety of manifestations, which might make the diagnosis even more difficult. A delayed diagnosis can negatively affect the patient’s life. Therefore, there is an urgent need to increase the scientific and medical knowledge about rare diseases. Natural Language Processing (NLP) and Deep Learning can help to extract relevant information about rare diseases to facilitate their diagnosis and treatments.MethodsThe paper explores several deep learning techniques such as Bidirectional Long Short Term Memory (BiLSTM) networks or deep contextualized word representations based on Bidirectional Encoder Representations from Transformers (BERT) to recognize rare diseases and their clinical manifestations (signs and symptoms).ResultsBioBERT, a domain-specific language representation based on BERT and trained on biomedical corpora, obtains the best results with an F1 of 85.2% for rare diseases. Since many signs are usually described by complex noun phrases that involve the use of use of overlapped, nested and discontinuous entities, the model provides lower results with an F1 of 57.2%.ConclusionsWhile our results are promising, there is still much room for improvement, especially with respect to the identification of clinical manifestations (signs and symptoms).

A deep learning method DCWR with HANet for stock market prediction using news articles

Stock movement prediction is a challenging problem to analyze in both academic and financial research areas. The advancement of deep learning (DL) techniques has grasped the attention of researchers to employ them for predicting the stock market’s future trends. Few frameworks can understand the financial terms in literature, and the volatile nature of stock markets further complicates this process. This paper has tried to overcome the existing challenges by introducing a DL-based framework using financial news articles to forecast the stock market. After performing preprocessing step, the deep contextualized word representation (DCWR) approach is applied for feature extraction. In the next step, the independent component analysis (ICA) method is used for feature reduction. Finally, the resultant features train the hierarchical attention networks (HANet) classifier to predict the stock movements. The proposed scheme is evaluated over the 7 years of data from a publicly available dataset gathered from the Reuter’s website and attained an average prediction accuracy of 92.5% which shows our framework’s robustness.

Investigating Maps of Science Using Contextual Proximity of Citations Based on Deep Contextualized Word Representation

The citation intent extraction and classification has long been studied as it is a good measure of relevancy. Different approaches have classified the citations into different classes; including weak and strong, positive and negative, important and unimportant. Others have gone further from binary classification to multi-classes, including extension, use, background, or comparison. Researchers have utilized various elements of the information, including both meta and contents of the paper. The actual context of any referred article lies within the citation context where a paper is referred. Various attempts have been made to study the citation context to capture the citation intent, but very few have encoded the words to their contextual representations. For automated classification, we need to train deep learning models, which take the citation context as input and provides the reason for citing a paper. Deep neural models work on numeric data, and therefore, we must convert the text information to its numeric representation. Natural languages are much complex than computer languages. Computer languages have a pre-defined fixed syntax where each word has a unique meaning. In contrast, every word in natural language may have a different meaning and may well be understood by understanding the position, previous discussion, and neighboring words. The extra information provides the context of a word within a sentence. We have, therefore, used contextual word representation, which is trained through deep neural networks. Deep models require massive data for generalizing the model, however, the existing state-of-the-art datasets don’t provide much information for the training models to get generalized. Therefore, we have developed our own scholarly dataset, Citation Context Dataset with Intent (C2D-I), an extension of the C2D dataset. We used a transformers based model for capturing the contextual representation of words. Our proposed method outperformed the existing benchmark methods with F1 score of 89%.

Prediction of user loyalty in mobile applications using deep contextualized word representations

ABSTRACT Customer loyalty is important for many industries, including banking, telecommunications, gaming, and shopping, in terms of sustainability. In mobile applications, it is observed that the demand rises with the usage of mobile devices such as smartphones. Therefore, it is important to predict when players tend to leave an application. Most of the studies so far focus on churn prediction or customer loyalty in mobile applications by analyzing demographic, economic, and behavioral data about customers. In this work, we introduce sentiment analysis-based customer loyalty prediction in mobile applications using word embeddings, deep learning algorithms, and deep contextualized word representations. To our knowledge, this is the first study to evaluate loyalty of customers analyzing sentiments of users from their comments using deep learning, word embedding, and deep contextualized word representation models. For this purpose, CNNs, RNNs, LSTMs, BERT, MBERT, DistilBERT, RoBERT are used for classification purpose. On the other hand, word embedding models such as Word2Vec, GloVe, and FastText are employed for text representation. To demonstrate the impact of proposed model, comprehensive experiments are performed on seven different datasets. The experiment results show sentiment analysis of users in mobile applications can be a powerful indicator in terms of predicting customer loyalty.

Detecting formal thought disorder by deep contextualized word representations

Computational linguistics has enabled the introduction of objective tools that measure some of the symptoms of schizophrenia, including the coherence of speech associated with formal thought disorder (FTD). Our goal was to investigate whether neural network based utterance embeddings are more accurate in detecting FTD than models based on individual indicators. The present research used a comprehensive Embeddings from Language Models (ELMo) approach to represent interviews with patients suffering from schizophrenia (N=35) and with healthy people (N=35). We compared its results to the approach described by Bedi et al. (2015), referred to here as the coherence model. Evaluations were also performed by a clinician using the Scale for the Assessment of Thought, Language and Communication (TLC). Using all six TLC questions the ELMo obtained an accuracy of 80% in distinguishing patients from healthy people. Previously used coherence models were less accurate at 70%. The classifying clinician was accurate 74% of the time. Our analysis shows that both ELMo and TLC are sensitive to the symptoms of disorganization in patients. In this study methods using text representations from language models were more accurate than those based solely on the assessment of FTD, and can be used as measures of disordered language that complement human clinical ratings.

Deep Graph-Based Character-Level Chinese Dependency Parsing

Character-level Chinese dependency parsing has been a concern of several studies that naturally handle word segmentation, POS (Part of Speech) tagging and dependency parsing jointly in an end-to-end way. Previous work mostly concentrates on a transition-based framework for this task because of its easy adaption, which is extremely important when feature representation relies heavily on the decoding strategy, particularly under the traditional statistical setting. Recently, on the one hand, sophisticated deep neural networks and deep contextualized word representations have greatly weakened the dependence between feature representation and decoding. On the other hand, (first-order) graph-based models, especially the biaffine parsers, are straightforward for dependency parsing, and meanwhile they can yield competitive parsing performance. In this paper, we make a comprehensive investigation of the deep graph-based character-level dependency parsing for Chinese. We start from an extension of a standard graph-based biaffine parser, and then exploit Chinese BERT as well as our improved encoders based on transformers to enhance the character-level dependency parsing model. We conduct a series of experiments on the Chinese benchmark datasets, showing the performances of various graph-based character-level models and analyzing the advantages of the character-level dependency parsing under the deep neural setting.

Knowledge-enhanced biomedical named entity recognition and normalization: application to proteins and genes

BackgroundAutomated biomedical named entity recognition and normalization serves as the basis for many downstream applications in information management. However, this task is challenging due to name variations and entity ambiguity. A biomedical entity may have multiple variants and a variant could denote several different entity identifiers.ResultsTo remedy the above issues, we present a novel knowledge-enhanced system for protein/gene named entity recognition (PNER) and normalization (PNEN). On one hand, a large amount of entity name knowledge extracted from biomedical knowledge bases is used to recognize more entity variants. On the other hand, structural knowledge of entities is extracted and encoded as identifier (ID) embeddings, which are then used for better entity normalization. Moreover, deep contextualized word representations generated by pre-trained language models are also incorporated into our knowledge-enhanced system for modeling multi-sense information of entities. Experimental results on the BioCreative VI Bio-ID corpus show that our proposed knowledge-enhanced system achieves 0.871 F1-score for PNER and 0.445 F1-score for PNEN, respectively, leading to a new state-of-the-art performance.ConclusionsWe propose a knowledge-enhanced system that combines both entity knowledge and deep contextualized word representations. Comparison results show that entity knowledge is beneficial to the PNER and PNEN task and can be well combined with contextualized information in our system for further improvement.

Hybrid Deep Neural Network-Based Text Representation Model to Improve Microblog Retrieval

Retrieving relevant information from Twitter is always a challenging task given its vocabulary mismatch, sheer volume and noise. Representing the content of text tweets is a critical part of any microblog retrieval model. For this reason, deep neural networks can be used for learning good representations of text data and then conduct to a better matching. In this paper, we are interested in improving both representation and retrieval effectiveness in microblogs. For that, a Hybrid-Deep Neural-Network-based text representation model is proposed to extract effective features’ representations for clustering oriented microblog retrieval. HDNN combines recurrent neural network and feedforward neural network architectures. Specifically, using a bi-directional LSTM, we first generate a deep contextualized word representation which incorporates character n-grams form FasText. However, these contextual embedded existing in a high-dimensional space are not all important. Some of them are redundant, correlated and sometimes noisy making the learning models over-fitting, complex and less interpretable. To deal with these problems, we proposed a Hybrid-Regularized-Autoencoder-based method which combines autoencoder with Elastic Net regularization for an effective unsupervised feature selection and extraction. Our experimental results show that the performance of clustering and especially information retrieval in microblogs depend heavily on features’ representation.

Part-of-Speech Tagging Using Multiview Learning

In natural language processing, character-level representations are vector representations of the particular character. Character-level representations have recently focused on enriching subword information by stacking deep neural models. Ideally, applications of several character-level representations can help capture different aspects of the subword information. However, this approach has often failed in the past, mainly because of the nature of traditionally used simple concatenation models. In this study, we explore different character-level modeling techniques. During the learning process, long short-term memory-based character representations can introduce different views for a part-of-speech tagger. After investigating two previously reported techniques, we propose two additional extended methods: (1) a multihead-attention character-level representation for capturing several aspects of subword information, and (2) an optimal structure for training two different character-level embeddings based on joint learning. We evaluate our results on the part-of-speech (POS) tagging dataset of the Conference on Natural Language Learning (CoNLL) 2018 shared task in universal dependencies. We show that our method substantially improves POS tagging results for many morphologically rich languages where the character information should be considered more substantially. Moreover, we compare the performance of our model with recent state-of-the-art POS taggers, which are trained with language models such as Bidirectional Encoder Representations from Transformers (BERT) and Deep Contextualized Word Representations (ELMo); our multiview tagger shows better results for nine languages. The proposed character model shows significant improvements in Ancient Greek, with average gains of 8.89 points in accuracy compared to the previous word representation model. Therefore, our empirical experiments indicate that character-level representations are more important than word representations for morphologically rich languages in terms of performance.

An Empirical Test of GRUs and Deep Contextualized Word Representations on De-Identification.

De-identification aims to remove 18 categories of protected health information from electronic health records. Ideally, de-identification systems should be reliable and generalizable. Previous research has focused on improving performance but has not examined generalizability. This paper investigates both performance and generalizability. To improve current state-of-the-art performance based on long short-term memory (LSTM) units, we introduce a system that uses gated recurrent units (GRUs) and deep contextualized word representations, both of which have never been applied to de-identification. We measure performance and generalizability of each system using the 2014 i2b2/UTHealth and 2016 CEGS N-GRID de-identification datasets. We show that deep contextualized word representations improve state-of-the-art performance, while the benefit of switching LSTM units with GRUs is not significant. The generalizability of de-identification system significantly improved with deep contextualized word representations; in addition, LSTM units-based system is more generalizable than the GRUs-based system.

A Deep Learning Approach With Deep Contextualized Word Representations for Chemical–Protein Interaction Extraction From Biomedical Literature

Mining interactions between chemicals and proteins/genes is of crucial relevance for clinical medicine, adverse drug effects, and pharmacological research. Although chemical-protein interactions (CPIs) can be manually extracted, this process is expensive and time-consuming. Therefore, it is of considerable significance to automatically extract CPIs from biomedical literature. Currently, the popular methods for CPI extraction are based on deep learning to avoid sophisticated handcrafted features derived from linguistic analyses. However, the performance of existing methods is usually unsatisfactory. The reasons may be that (1) traditional word-embedding methods cannot adequately model context information, and (2) it is difficult to effectively distinguish which words play critical roles in long biomedical sentences. In this study, we propose a novel Deep-contextualized Stacked Bi-LSTM model (DS-LSTM) to tackle the drawbacks of existing methods. Specifically, our model mainly consists of three components: deep contextualized word representations, the entity attention mechanism, and stacked bidirectional long short-term memory networks (Bi-LSTMs). The deep contextualized word representations are introduced to effectively model complex characteristics of word use (e.g., syntax and semantics) and the variations of these words in the context (i.e., to model polysemy), thereby generating context information. The entity attention mechanism is applied to prioritize the weights of words associated with target entities to distinguish which words play critical roles in long biomedical sentences. We evaluate our model on the CHEMPROT corpus. Our approach achieves a micro-averaged F-score of 69.44%, which is significantly higher than existing state-of-the-art methods. Experimental results show that our approach can adequately model context information, effectively distinguish which words play critical roles in long biomedical sentences and, therefore, improve the overall performance.

R-Trans: RNN Transformer Network for Chinese Machine Reading Comprehension

Machine reading comprehension (MRC) has gained increasingly wide attention over the past few years. A variety of benchmark datasets have been released, which triggers the development of quite a few MRC approaches based on deep learning techniques. However, most existing models are designed to address English MRC. When applying them directly to Chinese documents, the performance often degrades considerably because of some special characteristics of Chinese, the inevitable segmentation errors in particular. In this paper, we present the RNN Transformer network to tackle the Chinese MRC task. To mitigate the influence of incorrect word segmentation and mine sequential information of whole sentences, deep contextualized word representations and bidirectional gated recurrent units networks are adopted in our model. The extensive experiments have been conducted on a very large scale Chinese MRC corpus, viz., the Les MMRC dataset. The results show that the proposed model outperforms the baseline and other prevalent MRC models notably, and established a new state-of-the-art record on the Les MMRC dataset.

Language Model-Driven Topic Clustering and Summarization for News Articles

Topic models have been widely utilized in Topic Detection and Tracking tasks, which aim to detect, track, and describe topics from a stream of broadcast news reports. However, most existing topic models neglect semantic or syntactic information and lack readable topic descriptions. To exploit semantic and syntactic information, Language Models (LMs) have been applied in many supervised NLP tasks. However, there are still no extensions of LMs for unsupervised topic clustering. Moreover, it is difficult to employ general LMs (e.g., BERT) to produce readable topic summaries due to the mismatch between the pretraining method and the summarization task. In this paper, noticing the similarity between content and summary, first we propose a Language Model-based Topic Model (LMTM) for Topic Clustering by using an LM to generate a deep contextualized word representation. Then, a new method of training a Topic Summarization Model is introduced, where it is not only able to produce brief topic summaries but also used as an LM in LMTM for topic clustering. Empirical evaluations of two different datasets show that the proposed LMTM method achieves better performance over four baselines for JC, FMI, precision, recall and F1-score. Additionally, the generated readable and reasonable summaries also validate the rationality of our model components.