Word-level Embeddings Research Articles

Combination of domain knowledge and deep learning for sentiment analysis of short and informal messages on social media

Sentiment analysis has been emerging recently as one of major natural language processing (NLP) tasks, with the increasing significance of social media channels for brands to observe user opinions about their products. In the previous work, we proposed to combine the typical deep learning techniques with domain knowledge. The combination is used for acquiring additional training data augmentation and a more reasonable loss function. However, there is a high volume of short and informal messages posted by users which makes the existing works suffer from many difficulties. In this work, we further improve our architecture, aiming to handle those problems, by various substantial enhancements, including negation-based data augmentation, transfer learning for word embeddings, combination of word-level embeddings and character-level embeddings, and using multitask learning technique for attaching domain knowledge rules in the learning process. Those enhancements help us to enjoy significant improvement in performance once experimenting on real datasets.

Combination of domain knowledge and deep learning for sentiment analysis of short and informal messages on social media

Sentiment analysis has been emerging recently as one of the major natural language processing (NLP) tasks in many applications. Especially, as social media channels (e.g. social networks or forums) have become significant sources for brands to observe user opinions about their products, this task is thus increasingly crucial. However, when applied with real data obtained from social media, we notice that there is a high volume of short and informal messages posted by users on those channels. This kind of data makes the existing works suffer from many difficulties to handle, especially ones using deep learning approaches. In this paper, we propose an approach to handle this problem. This work is extended from our previous work, in which we proposed to combine the typical deep learning technique of Convolutional Neural Networks with domain knowledge. The combination is used for acquiring additional training data augmentation and a more reasonable loss function. In this work, we further improve our architecture by various substantial enhancements, including negation-based data augmentation, transfer learning for word embeddings, the combination of word-level embeddings and character-level embeddings, and using multitask learning technique for attaching domain knowledge rules in the learning process. Those enhancements, specifically aiming to handle short and informal messages, help us to enjoy significant improvement in performance once experimenting on real datasets.

Boosting Arabic Named-Entity Recognition With Multi-Attention Layer

Sequence labeling models with recurrent neural network variants, such as long short-term memory (LSTM) and gated recurrent unit (GRU), show promising performance on several natural language processing (NLP) problems, including named-entity recognition (NER). Most existing models utilize word embeddings for capturing similarities between words. However, they lag when handling previously unobserved or infrequent words. Moreover, the attention mechanism has been used to improve sequence labeling tasks. In this paper, we propose an efficient multi-attention layer system for the Arabic named-entity recognition (ANER) task. In addition to word-level embeddings, we adopt character-level embeddings and combine them via an embedding-level attention mechanism. The output is fed into an encoder unit with bidirectional-LSTM, followed by another self-attention layer that is used to boost the system performance. Our model achieves approximately matched F1 score of 91% on the “ANERCorpus.” The overall experimental results demonstrate that our method is superior to other systems. Our approach using multi-layer attention mechanism yields a new state-of-the-art result for the ANER.

Chinese Event Extraction Based on Attention and Semantic Features: A Bidirectional Circular Neural Network

Chinese event extraction uses word embedding to capture similarity, but suffers when handling previously unseen or rare words. From the test, we know that characters may provide some information that we cannot obtain in words, so we propose a novel architecture for combining word representations: character–word embedding based on attention and semantic features. By using an attention mechanism, our method is able to dynamically decide how much information to use from word or character level embedding. With the semantic feature, we can obtain some more information about a word from the sentence. We evaluate different methods on the CEC Corpus, and this method is found to improve performance.

Chinese Text Sentiment Analysis using Bilinear Character-Word Convolutional Neural Networks

Text Sentiment Analysis (TSA) is becoming a hot area of research in the field of Natural Language Processing (NLP). There are many researches on English text sentiment analysis, while the Chinese do not attract sufficient attention. In this paper, we provide a novel deep learning model called Bilinear Character-Word Convolutional Neural Networks (BCWCNN) to deal with Chinese text sentiment analysis task. Our model represent a Chinese sentence as a bilinear combination of features learned from two-stream CNN models, which receives character-level and word-level embedding features as input, respectively. Experiments conducted on Chinese text sentiment corpus demonstrate that the proposed architecture significantly improve the performance of this task compared to other existing architectures.

An Attentive Neural Sequence Labeling Model for Adverse Drug Reactions Mentions Extraction

Adverse drug reactions (ADRs) are a main cause of morbidity and mortality in patients. Extracting mentions of ADRs from the health-related text has important applications in biomedical research. Existing work mainly utilizes feature-based pipeline methods or neural network models that use only word embeddings as input features. These methods either require many efforts to design task-specific features or suffer misclassification on those words, which have not been seen before. Therefore, we propose an end-to-end neural sequence labeling model that labels words in an input sequence with ADRs membership tags. In addition to word-level embeddings, we also adopt character-level embeddings and combine them via an embedding-level attention mechanism. Through such an attention mechanism, our model can dynamically determine how much information to utilize from a word- or character-level component. In addition, we use the intermediate output of the model as an auxiliary classifier and combine it with the final output of the model to improve the overall performance. We evaluate different architectures on two ADRs labeling datasets. One is an ADRs-related Twitter corpus that includes many informal vocabularies and irregular grammar, and the other is a biomedical text extracted from PubMed abstracts with many professional terms and technical descriptions. Our model achieves approximate match F1 scores of 0.844 and 0.906 for ADRs identification on the Twitter and PubMed datasets, respectively. It presents the state-of-the-art performance on both the datasets. Our system is completely end-to-end, requires no task-specific feature engineering or hand-crafted features, and thus can be generalized to a wide range of sequence labeling tasks.

Disease named entity recognition from biomedical literature using a novel convolutional neural network

BackgroundAutomatic disease named entity recognition (DNER) is of utmost importance for development of more sophisticated BioNLP tools. However, most conventional CRF based DNER systems rely on well-designed features whose selection is labor intensive and time-consuming. Though most deep learning methods can solve NER problems with little feature engineering, they employ additional CRF layer to capture the correlation information between labels in neighborhoods which makes them much complicated.MethodsIn this paper, we propose a novel multiple label convolutional neural network (MCNN) based disease NER approach. In this approach, instead of the CRF layer, a multiple label strategy (MLS) first introduced by us, is employed. First, the character-level embedding, word-level embedding and lexicon feature embedding are concatenated. Then several convolutional layers are stacked over the concatenated embedding. Finally, MLS strategy is applied to the output layer to capture the correlation information between neighboring labels.ResultsAs shown by the experimental results, MCNN can achieve the state-of-the-art performance on both NCBI and CDR corpora.ConclusionsThe proposed MCNN based disease NER method achieves the state-of-the-art performance with little feature engineering. And the experimental results show the MLS strategy’s effectiveness of capturing the correlation information between labels in the neighborhood.

Engineering doc2vec for automatic classification of product descriptions on O2O applications

In this paper, we develop an automatic product classifier that can become a vital part of a natural user interface for an integrated online-to-offline (O2O) service platform. We devise a novel feature extraction technique to represent product descriptions that are expressed in full natural language sentences. We specifically adapt doc2vec algorithm that implements the document embedding technique. Doc2vec is a way to predict a vector of salient contexts that are specific to a document. Our classifier is trained to classify a product description based on the doc2vec-based feature that is augmented in various ways. We trained and tested our classifier with up to 53,000 real product descriptions from Groupon, a popular social commerce site that also offers O2O commerce features such as online ordering for in-store pick-up. Compared to the baseline approaches of using bag-of-words modeling and word-level embedding, our classifier showed significant improvement in terms of classification accuracy when our adapted doc2vec-based feature was used.

Definition Modeling: Learning to Define Word Embeddings in Natural Language

Distributed representations of words have been shown to capture lexical semantics, based on their effectiveness in word similarity and analogical relation tasks. But, these tasks only evaluate lexical semantics indirectly. In this paper, we study whether it is possible to utilize distributed representations to generate dictionary definitions of words, as a more direct and transparent representation of the embeddings' semantics. We introduce definition modeling, the task of generating a definition for a given word and its embedding. We present different definition model architectures based on recurrent neural networks, and experiment with the models over multiple data sets. Our results show that a model that controls dependencies between the word being defined and the definition words performs significantly better, and that a character-level convolution layer that leverages morphology can complement word-level embeddings. Our analysis reveals which components of our models contribute to accuracy. Finally, the errors made by a definition model may provide insight into the shortcomings of word embeddings.