Distributed Word Representations Research Articles

Nonparametric Spherical Topic Modeling with Word Embeddings.

Traditional topic models do not account for semantic regularities in language. Recent distributional representations of words exhibit semantic consistency over directional metrics such as cosine similarity. However, neither categorical nor Gaussian observational distributions used in existing topic models are appropriate to leverage such correlations. In this paper, we propose to use the von Mises-Fisher distribution to model the density of words over a unit sphere. Such a representation is well-suited for directional data. We use a Hierarchical Dirichlet Process for our base topic model and propose an efficient inference algorithm based on Stochastic Variational Inference. This model enables us to naturally exploit the semantic structures of word embeddings while flexibly discovering the number of topics. Experiments demonstrate that our method outperforms competitive approaches in terms of topic coherence on two different text corpora while offering efficient inference.

Bilingual distributed word representations from document-aligned comparable data

We propose a new model for learning bilingual word representations from nonparallel document-aligned data. Following the recent advances in word representation learning, our model learns dense real...

An Unsupervised Graph Based Continuous Word Representation Method for Biomedical Text Mining.

In biomedical text mining tasks, distributed word representation has succeeded in capturing semantic regularities, but most of them are shallow-window based models, which are not sufficient for expressing the meaning of words. To represent words using deeper information, we make explicit the semantic regularity to emerge in word relations, including dependency relations and context relations, and propose a novel architecture for computing continuous vector representation by leveraging those relations. The performance of our model is measured on word analogy task and Protein-Protein Interaction Extraction (PPIE) task. Experimental results show that our method performs overall better than other word representation models on word analogy task and have many advantages on biomedical text mining.

KNET

Neural network techniques are widely applied to obtain high-quality distributed representations of words (i.e., word embeddings) to address text mining, information retrieval, and natural language processing tasks. Most recent efforts have proposed several efficient methods to learn word embeddings from context such that they can encode both semantic and syntactic relationships between words. However, it is quite challenging to handle unseen or rare words with insufficient context. Inspired by the study on the word recognition process in cognitive psychology, in this article, we propose to take advantage of seemingly less obvious but essentially important morphological knowledge to address these challenges. In particular, we introduce a novel neural network architecture called KNET that leverages both words’ contextual information and morphological knowledge to learn word embeddings. Meanwhile, this new learning architecture is also able to benefit from noisy knowledge and balance between contextual information and morphological knowledge. Experiments on an analogical reasoning task and a word similarity task both demonstrate that the proposed KNET framework can greatly enhance the effectiveness of word embeddings.

Incorporating linguistic knowledge for learning distributed word representations.

Combined with neural language models, distributed word representations achieve significant advantages in computational linguistics and text mining. Most existing models estimate distributed word vectors from large-scale data in an unsupervised fashion, which, however, do not take rich linguistic knowledge into consideration. Linguistic knowledge can be represented as either link-based knowledge or preference-based knowledge, and we propose knowledge regularized word representation models (KRWR) to incorporate these prior knowledge for learning distributed word representations. Experiment results demonstrate that our estimated word representation achieves better performance in task of semantic relatedness ranking. This indicates that our methods can efficiently encode both prior knowledge from knowledge bases and statistical knowledge from large-scale text corpora into a unified word representation model, which will benefit many tasks in text mining.

Product Name Recognition for Informal Text: Exploring Features

Product name recognition refers to locate the name of product in text automatically. We built a conditional random field to recognize product name from forum posts and explored various features to compare their impact on the performance. These features include not only traditional features used for NER but also distributed word representations which are novel features obtained from new area of machine learning.

Word frequency and the lateralization of lexical processes

Despite the extensive clinical and neuropsychological evidence for major differences in language function between the left and right hemispheres, it has proven difficult to demonstrate reliable differences in lexical processing. It is particularly surprising that frequency has been found to have no differential effects upon word recognition in the hemispheres, in view of the evidence that this variable has significant effects upon lexical processing. It is possible, however, that any interaction between word frequency and hemispheric processes is too subtle or too complex to be revealed by the simple dichotomous studies previously attempted. This study manipulated word frequency across seven levels in an effort to provide a fine-grained insight into the relationship between word frequency and lexical processing in the hemispheres. The results revealed a clear additivity between visual field and word frequency across all levels of frequency, for both RT and errors. Such findings create difficulties for the interpretation of studies of lateralized lexical access in general, and are particularly hard to reconcile with lateral asymmetries revealed in other aspects of lexical processing. It is proposed that a resolution of this problem may depend upon the development of theories that emphasise the notion of laterally distributed word representations, rather than discrete lexical structures on each side of the brain.