Distributed Representation Model Research Articles

조선교육요람 영문판(1913년, 1920년)에 나타난 교육기관 연구: 단어 출현빈도 분석과 분산표상 모델을 활용하여

조선교육요람 영문판(1913년, 1920년)에 나타난 교육기관 연구: 단어 출현빈도 분석과 분산표상 모델을 활용하여

Multi-level out-of-vocabulary words handling approach

Distributed representation models can generate a vector representation only for words that belong to a finite vocabulary collected from the training data. If out-of-vocabulary (OOV) words are not handled properly, they can impair the performance of machine learning methods in a given natural language processing task. This study offers a new methodology based on the consolidated top-down human reading theory, which may serve as a strong basis for developing new techniques to deal with the OOV problem. For this, we present MLOH, a Multi-Level OOV Handling approach, based on three chained strategies: analogy, decoding, and prediction. The techniques available in the literature, in general, are limited since they often resolve specific types of OOV words, such as those that can be inferred by analyzing their morphological structure or context. Compared to the process used by human readers to infer unknown words, using a single strategy is generally not effective. We evaluated MLOH performance on tasks that can be highly affected by OOV words, such as part-of-speech tagging, named entity recognition, and text categorization of short and noisy texts. The results indicate that the proposed approach is promising since it could handle most of the OOV words presented, is more generalist, and obtained competitive performance in all experiments.

DeepD2V: A Novel Deep Learning-Based Framework for Predicting Transcription Factor Binding Sites from Combined DNA Sequence.

Predicting in vivo protein–DNA binding sites is a challenging but pressing task in a variety of fields like drug design and development. Most promoters contain a number of transcription factor (TF) binding sites, but only a small minority has been identified by biochemical experiments that are time-consuming and laborious. To tackle this challenge, many computational methods have been proposed to predict TF binding sites from DNA sequence. Although previous methods have achieved remarkable performance in the prediction of protein–DNA interactions, there is still considerable room for improvement. In this paper, we present a hybrid deep learning framework, termed DeepD2V, for transcription factor binding sites prediction. First, we construct the input matrix with an original DNA sequence and its three kinds of variant sequences, including its inverse, complementary, and complementary inverse sequence. A sliding window of size k with a specific stride is used to obtain its k-mer representation of input sequences. Next, we use word2vec to obtain a pre-trained k-mer word distributed representation model. Finally, the probability of protein–DNA binding is predicted by using the recurrent and convolutional neural network. The experiment results on 50 public ChIP-seq benchmark datasets demonstrate the superior performance and robustness of DeepD2V. Moreover, we verify that the performance of DeepD2V using word2vec-based k-mer distributed representation is better than one-hot encoding, and the integrated framework of both convolutional neural network (CNN) and bidirectional LSTM (bi-LSTM) outperforms CNN or the bi-LSTM model when used alone. The source code of DeepD2V is available at the github repository.

A novel recommendation algorithm with knowledge graph

Personalized recommendation is an important topic in recommendation algorithm research. Traditional collaborative filtering recommendation algorithms suffer the sparseness and cold start problems. Existing knowledge graph based recommendation algorithms miss the high-order similarity at the subgraph level. This paper introduces a RNN-based distributed representation model of knowledge graph called KG-GRU, which uses a sequence containing nodes and relationships to model the subgraph similarity in the same embedded vector space. Then, a personalized recommendation algorithm based on knowledge graph called KG-GRU4Rec, is proposed. KG-GRU4Rec implements an end-to-end predictive user rating model. Experimental results demonstrate that KG-GRU learns more accurate knowledge about entities and relationships in the graph and KG-GRU4Rec outperforms the comparison algorithm in both hit ratio and average reciprocal ranking.

SpliceVec: Distributed feature representations for splice junction prediction

Identification of intron boundaries, called splice junctions, is an important part of delineating gene structure and functions. This also provides valuable insights into the role of alternative splicing in increasing functional diversity of genes. Identification of splice junctions through RNA-seq is by mapping short reads to the reference genome which is prone to errors due to random sequence matches. This encourages identification of splicing junctions through computational methods based on machine learning. Existing models are dependent on feature extraction and selection for capturing splicing signals lying in the vicinity of splice junctions. But such manually extracted features are not exhaustive. We introduce distributed feature representation, SpliceVec, to avoid explicit and biased feature extraction generally adopted for such tasks. SpliceVec is based on two widely used distributed representation models in natural language processing. Learned feature representation in form of SpliceVec is fed to multilayer perceptron for splice junction classification task. An intrinsic evaluation of SpliceVec indicates that it is able to group true and false sites distinctly. Our study on optimal context to be considered for feature extraction indicates inclusion of entire intronic sequence to be better than flanking upstream and downstream region around splice junctions. Further, SpliceVec is invariant to canonical and non-canonical splice junction detection. The proposed model is consistent in its performance even with reduced dataset and class-imbalanced dataset. SpliceVec is computationally efficient and can be trained with user-defined data as well.

Application of the distributed document representation in the authorship attribution task for small corpora

Distributed word representation in a vector space (word embeddings) is a novel technique that allows to represent words in terms of the elements in the neighborhood. Distributed representations can be extended to larger language structures like phrases, sentences, paragraphs and documents. The capability to encode semantic information of texts and the ability to handle high- dimensional datasets are the reasons why this representation is widely used in various natural language processing tasks such as text summarization, sentiment analysis and syntactic parsing. In this paper, we propose to use the distributed representation at the document level to solve the task of the authorship attribution. The proposed method learns distributed vector representations at the document level and then uses the SVM classifier to perform the automatic authorship attribution. We also propose to use the word n-grams (instead of the words) as the input data type for learning the distributed representation model. We conducted experiments over six datasets used in the state-of-the-art works, and for the majority of the datasets, we obtained comparable or better results. Our best results were obtained using the combination of words and n-grams of words as the input data types. Training data are relatively scarce, which did not affect the distributed representation.

Using Deep Learning for Compound Selectivity Prediction.

Compound selectivity prediction plays an important role in identifying potential compounds that bind to the target of interest with high affinity. However, there is still short of efficient and accurate computational approaches to analyze and predict compound selectivity. In this paper, we propose two methods to improve the compound selectivity prediction. We employ an improved multitask learning method in Neural Networks (NNs), which not only incorporates both activity and selectivity for other targets, but also uses a probabilistic classifier with a logistic regression. We further improve the compound selectivity prediction by using the multitask learning method in Deep Belief Networks (DBNs) which can build a distributed representation model and improve the generalization of the shared tasks. In addition, we assign different weights to the auxiliary tasks that are related to the primary selectivity prediction task. In contrast to other related work, our methods greatly improve the accuracy of the compound selectivity prediction, in particular, using the multitask learning in DBNs with modified weights obtains the best performance.

The priming effect of translation equivalents across languages for concrete and abstract words

The present study used a masked priming paradigm and two language tasks (lexical decision, semantic categorical judgment) to investigate whether concrete and abstract words share the same degree of conceptual representation across languages for bilinguals. The results showed that the priming effect of translation equivalents did not differ for concrete and abstract words in the lexical decision task, in both prime-target directions (in Experiment 1). The same results were also found in the semantic categorical judgment task in either prime-target direction (in Experiment 2). Our results do not provide support for the representation difference hypothesis of concrete and abstract words of Distributed Representation Model (De Groot, 1992a, 1992b; Van Hell & De Groot, 1998), which assumes that concrete words share more semantic components in the conceptual representations across languages, compared with abstract words. Rather, our findings suggest that both concrete and abstract words have the same degree of overlap in conceptual representations across a bilingual's two languages.

Effect of multiple translations and cognate status on translation recognition performance of balanced bilinguals

When participants are asked to translate an ambiguous word, they are slower and less accurate than in the case of single-translation words (e.g., Láxen & Lavour, 2010; Tokowicz & Kroll, 2007). We report an experiment to further examine this multiple-translation effect by investigating the influence of variables shown to be relevant in bilingual processing. The experiment included cognates and non-cognates with one translation or with multiple translations. The latter were presented with their dominant or subordinate translations. Highly-proficient balanced bilinguals responded to a translation recognition task in the two language directions (Catalan–Spanish and Spanish–Catalan). The results showed a significant multiple-translation effect in both cognates and non-cognates. Moreover, this effect was obtained regardless of language dominance and translation direction. Participants were faster and more accurate when performing translation recognition for the dominant than for the subordinate translations. The findings are interpreted adopting the Distributed Representation Model (de Groot, 1992b).

Sexual Imprinting in Zebra Finch Males: a Differential Effect of Successive and Simultaneous Experience With Two Colour Morphs

Abstract Recent developments in theories of sexual imprinting have emphasized that sexual imprinting may lead to a preference for partners that are slightly different from the stimuli an animal has originally been exposed to. Two mechanisms have been proposed to account for this phenomenon. 1. Separate representation model: This model assumes that as a result of imprinting an internal representation is formed of each individual which a bird experiences during early life. The imprinting process is thought to reduce the responsiveness to novel individuals, whereas at the same time a habituation process would have the effect that individuals resembling these representations too closely are refuted. 2. Mixed representation model: Young birds combine information about different individuals into one internal representation. As a consequence the later preference deviates from the original imprinting stimuli. There is evidence that zebra finch males use a number of stimuli for sexual imprinting, but it is unclear whether and how the way of exposure affects the internal representation. This was examined in zebra finch males that had either successive or simultaneous experience with two different colour morphs of the zebra finch. Each type of experience results in different effects on later preference. Successive experience leads to a preference for both colour morphs. That is, in multiple-stimuli tests with the imprinting morphs and intermediate colour morphs, males with successive experience showed a preference for both imprinting morphs. Simultaneous experience with both morphs does not lead to a double preference but to a preference for the colour morph of the mother. Comparison with the control groups revealed that males raised by mixed pairs had learned to suppress sexual responses to individuals of their father's morph. This implies that zebra finch males learn to discriminate male versus female conspecifics on morphological features during the time course of sexual imprinting. The results of neither type of exposure gave evidence for the formation of mixed representations, as was indicated by the lack of sexual interest which double imprinted males showed for intermediates between both morphs. It is suggested that the way information is stored during imprinting can better be viewed within the context of distributed representation models.