Distributed Word Representations Research Articles

Distributional semantic modeling: a revised technique to train term/word vector space models applying the ontology-related approach

We design a new technique for the distributional semantic modeling with a neural network-based approach to learn distributed term representations (or term embeddings) – term vector space models as a result, inspired by the recent ontology-related approach (using different types of contextual knowledge such as syntactic knowledge, terminological knowledge, semantic knowledge, etc.) to the identification of terms (term extraction) and relations between them (relation extraction) called semantic pre-processing technology – SPT. Our method relies on automatic term extraction from the natural language texts and subsequent formation of the problem-oriented or application-oriented (also deeply annotated) text corpora where the fundamental entity is the term (includes non-compositional and compositional terms). This gives us an opportunity to changeover from distributed word representations (or word embeddings) to distributed term representations (or term embeddings). The main practical result of our work is the development kit (set of toolkits represented as web service APIs and web application), which provides all necessary routines for the basic linguistic pre-processing and the semantic pre-processing of the natural language texts in Ukrainian for future training of term vector space models.

Towards syntax-aware token embeddings

AbstractDistributional semantic word representations are at the basis of most modern NLP systems. Their usefulness has been proven across various tasks, particularly as inputs to deep learning models. Beyond that, much work investigated fine-tuning the generic word embeddings to leverage linguistic knowledge from large lexical resources. Some work investigated context-dependent word token embeddings motivated by word sense disambiguation, using sequential context and large lexical resources. More recently, acknowledging the need for an in-context representation of words, some work leveraged information derived from language modelling and large amounts of data to induce contextualised representations. In this paper, we investigate Syntax-Aware word Token Embeddings (SATokE) as a way to explicitly encode specific information derived from the linguistic analysis of a sentence in vectors which are input to a deep learning model. We propose an efficient unsupervised learning algorithm based on tensor factorisation for computing these token embeddings given an arbitrary graph of linguistic structure. Applying this method to syntactic dependency structures, we investigate the usefulness of such token representations as part of deep learning models of text understanding. We encode a sentence either by learning embeddings for its tokens and the relations between them from scratch or by leveraging pre-trained relation embeddings to infer token representations. Given sufficient data, the former is slightly more accurate than the latter, yet both provide more informative token embeddings than standard word representations, even when the word representations have been learned on the same type of context from larger corpora (namely pre-trained dependency-based word embeddings). We use a large set of supervised tasks and two major deep learning families of models for sentence understanding to evaluate our proposal. We empirically demonstrate the superiority of the token representations compared to popular distributional representations of words for various sentence and sentence pair classification tasks.

MimicProp: Learning to Incorporate Lexicon Knowledge into Distributed Word Representation for Social Media Analysis

Lexicon-based methods and word embeddings are the two widely used approaches for analyzing texts in social media. The choice of an approach can have a significant impact on the reliability of the text analysis. For example, lexicons provide manually curated, domain-specific attributes about a limited set of words, while word embeddings learn to encode some loose semantic interpretations for a much broader set of words. Text analysis can benefit from a representation that offers both the broad coverage of word embeddings and the domain knowledge of lexicons. This paper presents MimicProp, a new graph-mode method that learns a lexicon-aligned word embedding. Our approach improves over prior graph-based methods in terms of its interpretability (i.e., lexicon attributes can be recovered) and generalizability (i.e., new words can be learned to incorporate lexicon knowledge). It also effectively improves the performance of downstream analysis applications, such as text classification.

Multi-source social media data sentiment analysis using bidirectional recurrent convolutional neural networks

Subjectivity detection in the text is essential for sentiment analysis, which requires many techniques to perceive unanticipated means of communication. Few accomplishments adapted to capture the syntactic, semantic, and contextual sentimental information via distributed word representations (DWRs)11Distributed work representations .. This paper, concatenating the DWRs through a weighted mechanism on Recurrent Neural Network (RNN) variants joint with Convolutional Neural network (CNN) distinctively involving weighted attentive pooling (WAP)22Weighted Attentive Pooling .. Whereas, CNNs with traditional pooling operations comprise many layers merely able to capture enough features. Our considerations empower the sentiment analysis over DWRs contains Word2vec, FastText, and GloVe to produce dense efficient concatenated representation (DECR)33Dense Efficient Concatenated Representation . to hold long term dependencies on a single RNN layer acquired by Parts of Speech Tagging (POS) explicitly with verbs, adverbs, and noun only. Then use these representations gained in a way, inputted to CNN contain single convolution layer engaging WAP on multi-source social media data to handle the issues of syntactic and semantic regularities as well as out of vocabulary (OOV) words. Experimentations demonstrate that DWRs together with proposed concatenation qualified in resolving the mentioned issues by moderate hyper-parameter configurations. Our architecture devoid of stacking multiple layers achieved modest accuracy of 89.67% by DECR-Bi-GRU-CNN (WAP) on IMDB as compared to random initialization 81.11% on SST.

Causally Denoise Word Embeddings Using Half-Sibling Regression

Distributional representations of words, also known as word vectors, have become crucial for modern natural language processing tasks due to their wide applications. Recently, a growing body of word vector postprocessing algorithm has emerged, aiming to render off-the-shelf word vectors even stronger. In line with these investigations, we introduce a novel word vector postprocessing scheme under a causal inference framework. Concretely, the postprocessing pipeline is realized by Half-Sibling Regression (HSR), which allows us to identify and remove confounding noise contained in word vectors. Compared to previous work, our proposed method has the advantages of interpretability and transparency due to its causal inference grounding. Evaluated on a battery of standard lexical-level evaluation tasks and downstream sentiment analysis tasks, our method reaches state-of-the-art performance.

Combining Distributed Word Representation and Document Distance for Short Text Document Clustering

This paper presents a method for clustering short text documents, such as news headlines, social media statuses, or instant messages. Due to the characteristics of these documents, which are usually short and sparse, an appropriate technique is required to discover hidden knowledge. The objective of this paper is to identify the combination of document representation, document distance, and document clustering that yields the best clustering quality. Document representations are expanded by external knowledge sources represented by a Distributed Representation. To cluster documents, a K-means partitioning-based clustering technique is applied, where the similarities of documents are measured by word mover’s distance. To validate the effectiveness of the proposed method, experiments were conducted to compare the clustering quality against several leading methods. The proposed method produced clusters of documents that resulted in higher precision, recall, F1- score, and adjusted Rand index for both real-world and standard data sets. Furthermore, manual inspection of the clustering results was conducted to observe the efficacy of the proposed method. The topics of each document cluster are undoubtedly reflected by members in the cluster.

A survey of state-of-the-art approaches for emotion recognition in text

Emotion recognition in text is an important natural language processing (NLP) task whose solution can benefit several applications in different fields, including data mining, e-learning, information filtering systems, human–computer interaction, and psychology. Explicit emotion recognition in text is the most addressed problem in the literature. The solution to this problem is mainly based on identifying keywords. Implicit emotion recognition is the most challenging problem to solve because such emotion is typically hidden within the text, and thus, its solution requires an understanding of the context. There are four main approaches for implicit emotion recognition in text: rule-based approaches, classical learning-based approaches, deep learning approaches, and hybrid approaches. In this paper, we critically survey the state-of-the-art research for explicit and implicit emotion recognition in text. We present the different approaches found in the literature, detail their main features, discuss their advantages and limitations, and compare them within tables. This study shows that hybrid approaches and learning-based approaches that utilize traditional text representation with distributed word representation outperform the other approaches on benchmark corpora. This paper also identifies the sets of features that lead to the best-performing approaches; highlights the impacts of simple NLP tasks, such as part-of-speech tagging and parsing, on the performances of these approaches; and indicates some open problems.

Unsupervised Word Embedding Learning by Incorporating Local and Global Contexts.

Word embedding has benefited a broad spectrum of text analysis tasks by learning distributed word representations to encode word semantics. Word representations are typically learned by modeling local contexts of words, assuming that words sharing similar surrounding words are semantically close. We argue that local contexts can only partially define word semantics in the unsupervised word embedding learning. Global contexts, referring to the broader semantic units, such as the document or paragraph where the word appears, can capture different aspects of word semantics and complement local contexts. We propose two simple yet effective unsupervised word embedding models that jointly model both local and global contexts to learn word representations. We provide theoretical interpretations of the proposed models to demonstrate how local and global contexts are jointly modeled, assuming a generative relationship between words and contexts. We conduct a thorough evaluation on a wide range of benchmark datasets. Our quantitative analysis and case study show that despite their simplicity, our two proposed models achieve superior performance on word similarity and text classification tasks.

Enhanced clustering models with wiki-based k-nearest neighbors-based representation for web search result clustering

Information retrieval is a difficult process due to the overabundance of information on the web. Nowadays, search result responds to user queries with too many results although only a few are relevant. Therefore, the existing clustering methods that fail in clustering snippets (short texts) of web documents due to the low frequencies of document terms should be deeply investigated. One of the approaches that can be used to solve this problem is the expansion of document terms with semantically similar terms. Hence, a list of terms with their closest and accurate semantically similar words (word representation) must be built. This study aims to design and develop a new framework to enhance the performance of web search result clustering (WSRC). The research also presents a new unsupervised distributed word representation scheme where each word is represented by a vector of its semantically related words; such as scheme expands snippets and user queries. The proposed framework consists of several activities, such as (1) various standard datasets (Open Directory Project [ODP]-239 and MORESQUE) that are used for evaluating search result clustering algorithms for most cited dataset works, (2) text pre-processing, (3) document representation based on a new wiki-based k-nearest neighbors (KNN) representation method, (4) effect of the proposed model on the performance of traditional clustering methods (k-means, k-medoids, single-linkage, and complete-linkage) for WSRC, and (5) evaluation stage of the proposed method. Results indicate that enhanced clustering methods, according to the new wiki-KNN based representation method in comparison with the baseline methods, show a significant improvement in WSRC. Furthermore, the new data representation scheme has enhanced the overall performance of clustering methods.

Social media sentiment analysis through parallel dilated convolutional neural network for smart city applications

Deep Learning is considered to leverage smart cities through social media sentiment analysis. The digital content in social media can be used for many smart city applications (SCAs)11Smart city applications.. Classical convolutional neural networks (CNNs) are challenging to parallelize and insufficient to capture long term contextual semantic features for sentiment analysis. In this perspective, this paper initially proposes a domain-specific distributed word representation (DS-DWR)22Domain Specific distributed word representation. with a considerably small corpus size induced from textual resources in social media. In DS-DWR, different Distributed Word Representations are concatenated to builds rich representations over the input sequence, which is worthwhile for infrequent and unseen terms. Second, a dilated convolutional neural network (D-CNN)33Dilated Convolutional Neural Network., which is composed of three parallel dilated convolutional neural network (PD-CNN)44Parallel dilated convolutional neural network. layers and a global average pooling (GAP)55Global Average Pooling. layer. Our considered parallel dilated convolution reduces dimension and incorporates an extension in the size of receptive fields without the loss of local information. Further, the long-term contextual semantic information is achieved by the use of different dilation rates. Experiments demonstrate that our architecture accomplishes comparable results with multiple hyperparameters tuning for better parallelism which leads to the minimized computational cost.

Time-sensitive clustering evolving textual data streams

Clustering a stream of text documents is an emerging subject of interest since it is widely used in analysing the content in social media and e-journals. The aim is to find a certain structure for unlabelled data based on a similarity criterion. However, few works have focused on this field and fall in this perspective, that's why a new document clustering approach adapted to a stream of text data and test it on news articles data sets is proposed. A distributed representation of words is used, and a bottom-up approach is used to represent documents as vectors on a unit hyper-sphere. The proposed approach gains its roots from the SPherical k-means (SPKM) algorithm and its underlying mixture of von-Mises Fisher (vMF) distributions. The proposed approach yields comparable results to baseline batch algorithm for stable data streams and superior results for rapidly evolving data streams.

The Ability of Word Embeddings to Capture Word Similarities

Distributed language representation has become the most widely used technique for language representation in various natural language processing tasks. Most of the natural language processing models that are based on deep learning techniques use already pre-trained distributed word representations, commonly called word embedding. Determining the most qualitative word embedding is of crucial importance for such models. However, selecting the appropriate word embedding is a perplexing task since the projected embedding space is not intuitive to humans.In this paper, we explore different approaches for creating distributed word representations. We perform an intrinsic evaluation of several state-of-the-art word embedding methods. Their performance on capturing word similarities is analysed with existing benchmark datasets for word pairs similarities. The research in this paper conducts a correlation analysis between ground truth word similarities and similarities obtained by different word embedding methods.

Improving Chinese Word Representation with Conceptual Semantics

The meaning of a word includes a conceptual meaning and a distributive meaning. Word embedding based on distribution suffers from insufficient conceptual semantic representation caused by data sparsity, especially for low-frequency words. In knowledge bases, manually annotated semantic knowledge is stable and the essential attributes of words are accurately denoted. In this paper, we propose a Conceptual Semantics Enhanced Word Representation (CEWR) model, computing the synset embedding and hypernym embedding of Chinese words based on the Tongyici Cilin thesaurus, and aggregating it with distributed word representation to have both distributed information and the conceptual meaning encoded in the representation of words. We evaluate the CEWR model on two tasks: word similarity computation and short text classification. The Spearman correlation between model results and human judgement are improved to 64.71%, 81.84%, and 85.16% on Wordsim297, MC30, and RG65, respectively. Moreover, CEWR improves the F1 score by 3% in the short text classification task. The experimental results show that CEWR can represent words in a more informative approach than distributed word embedding. This proves that conceptual semantics, especially hypernymous information, is a good complement to distributed word representation.

Reasoning over Arabic WordNet Relations with Neural Tensor Network

Arabic WordNet is an important resource for many tasks of natural language processing. However, it suffers from many problems. In this paper, we address the problem of the unseen relationships between words in Arabic WordNet. More precisely, we focus on the ability for new relationships to be learned ‘automatically’ in Arabic WordNet from existing relationships. Using the Neural Tensor Network, we investigate how it can be an advantageous technique to fill the relationship gaps between Arabic WordNet words. With minimum resources, this model delivers meaningful results. The critical component is how to represent the entities of Arabic WordNet. For that, we use AraVec, a set of pre-trained distributed word representation for the Arabic language. We show how much it helps to use these vectors for initialization. We evaluated the model, using a number of tests which reveal that semantically-initialized vectors provide considerable greater accuracy than randomly initialized ones.

Fine-grained neural decoding with distributed word representations

fMRI word decoding refers to decode what the human brain is thinking by interpreting functional Magnetic Resonance Imaging (fMRI) scans from people watching or listening to words, representing a sort of mind-reading technology. Existing works decoding words from imaging data have been largely limited to concrete nouns from a relatively small number of semantic categories. Moreover, such studies use different word-stimulus presentation paradigms and different computational models, lacking a comprehensive understanding of the influence of different factors on fMRI word decoding. In this paper, we present a large-scale evaluation of eight word embedding models and their combinations for decoding fine-grained fMRI data associated with three classes of words recorded from three stimulus-presentation paradigms. Specifically, we investigate the following research questions: (1) How does the brain-image decoder perform on different classes of words? (2) How does the brain-image decoder perform in different stimulus-presentation paradigms? (3) How well does each word embedding model allow us to decode neural activation patterns in the human brain? Furthermore, we analyze the most informative voxels associated with different classes of words, stimulus-presentation paradigms and word embedding models to explore their neural basis. The results have shown the following: (1) Different word classes can be decoded most effectively with different word embedding models. Concrete nouns and verbs are more easily distinguished than abstract nouns and verbs. (2) Among the three stimulus-presentation paradigms (picture, sentence and word clouds), the picture paradigm achieves the highest decoding accuracy, followed by the sentence paradigm. (3) Among the eight word embedding models, the model that encodes visual information obtains the best performance, followed by models that encode textual and contextual information. (4) Compared to concrete nouns, which activate mostly vision-related brain regions, abstract nouns activate broader brain regions such as the visual, language and default-mode networks. Moreover, both the picture paradigm and the model that encodes visual information have stronger associations with vision-related brain regions than other paradigms and word embedding models, respectively.

ArWordVec: efficient word embedding models for Arabic tweets

One of the major advances in artificial intelligence nowadays is to understand, process and utilize the humans’ natural language. This has been achieved by employing the different natural language processing (NLP) techniques along with the aid of the various deep learning approaches and architectures. Using the distributed word representations to substitute the traditional bag-of-words approach has been utilized very efficiently in the last years for many NLP tasks. In this paper, we present the detailed steps of building a set of efficient word embedding models called ArWordVec that are generated from a huge repository of Arabic tweets. In addition, a new method for measuring Arabic word similarity is introduced that has been used in evaluating the performance of the generated ArWordVec models. The experimental results show that the performance of the ArWordVec models overcomes the recently available models on Arabic Twitter data for the word similarity task. In addition, two of the large Arabic tweets datasets are used to examine the performance of the proposed models in the multi-class sentiment analysis task. The results show that the proposed models are very efficient and help in achieving a classification accuracy ratio exceeding 73.86% with a high average F1 value of 74.15.

What Do Language Representations Really Represent?

A neural language model trained on a text corpus can be used to induce distributed representations of words, such that similar words end up with similar representations. If the corpus is multilingual, the same model can be used to learn distributed representations of languages, such that similar languages end up with similar representations. We show that this holds even when the multilingual corpus has been translated into English, by picking up the faint signal left by the source languages. However, just as it is a thorny problem to separate semantic from syntactic similarity in word representations, it is not obvious what type of similarity is captured by language representations. We investigate correlations and causal relationships between language representations learned from translations on one hand, and genetic, geographical, and several levels of structural similarity between languages on the other. Of these, structural similarity is found to correlate most strongly with language representation similarity, whereas genetic relationships—a convenient benchmark used for evaluation in previous work—appears to be a confounding factor. Apart from implications about translation effects, we see this more generally as a case where NLP and linguistic typology can interact and benefit one another.

Cross-lingual plagiarism detection techniques for English-Hindi language pairs

Plagiarism is defined as stealing of another author’s language, thoughts, or ideas as one’s own original work. Most of the work in the literature has been emphasized on monolingual plagiarism detection. Given text in two different languages, the objective of Cross-Lingual Plagiarism Detection (CLPD) model is to determine if the text in one language is plagiarized with the text written in another language. The cross-lingual plagiarism detection is mainly an unexplored area and is very important to improve the ethical practice, academic integrity and honesty in academics. The problem of CLPD has emerged as very important yet difficult problem because it is very easy to steal the content of a document written in another language through translation softwares. In this paper, we present different cross-language plagiarism detection approaches in context of Indian languages pairs such as Hindi-English, English-Hindi language pairs. The research in cross-language plagiarism detection is very limited but has opened up opportunities due to development in recent advancement in distributed word representation techniques in monolingual and bi-lingual context.

BioWordVec,\xa0improving biomedical word embeddings with subword information and MeSH

Distributed word representations have become an essential foundation for biomedical natural language processing (BioNLP), text mining and information retrieval. Word embeddings are traditionally computed at the word level from a large corpus of unlabeled text, ignoring the information present in the internal structure of words or any information available in domain specific structured resources such as ontologies. However, such information holds potentials for greatly improving the quality of the word representation, as suggested in some recent studies in the general domain. Here we present BioWordVec: an open set of biomedical word vectors/embeddings that combines subword information from unlabeled biomedical text with a widely-used biomedical controlled vocabulary called Medical Subject Headings (MeSH). We assess both the validity and utility of our generated word embeddings over multiple NLP tasks in the biomedical domain. Our benchmarking results demonstrate that our word embeddings can result in significantly improved performance over the previous state of the art in those challenging tasks.

Concentration Areas of Sentiment Lexica in the Word Embedding Space

Sentiment lexicons play an important role in opinion mining systems and cognitive linguistics. Previous work aimed mostly at creating sentiment lexicons, but not thorough research into their fundamental properties. In this paper the arrangement of sentiment lexica in the multidimensional space of distributed word representations is studied. A hypothesis on the existence of sentiment lexica concentration areas is introduced and it is tested on the basis of the joint analysis of the distribution of sentiment words and general lexica. The results of the test allow to confirm the proposed hypothesis and discover the words which more than 80% of the sentiment lexica is concentrated around.