Dense Vector Research Articles

Semi-supervised network embedding with text information

Network embedding plays a pivotal role in network analysis, due to the capability of encoding each node to a low-dimensional dense feature vector. However, most existing network embedding approaches only focus on preserving structural information in the network. The text features and category attributes of nodes are ignored, which are important to network analysis. In this paper, we propose an innovative semi-supervised network embedding (SNE) model integrating structural information, text features and category attributes into embedding vectors simultaneously. Specifically, we design a structure preserving module and a text representation module to capture the global structural information and the text features separately. Meanwhile, a label indicator matrix and a supervised loss are proposed for preserving category information and mapping nodes in the same class closer. We utilize stacked auto-encoders to explore the highly nonlinear characteristics of the network. By optimizing the reconstruction loss and the designed supervised loss jointly in the proposed semi-supervised model, the embedding vectors are finally learned. Extensive experiments on real-world datasets demonstrate that our method is superior to the state-of-the-art baselines in a variety of tasks, including visualization, node classification and clustering.

Learning variable-length representation of words

A standard word embedding algorithm, such as ‘word2vec’, embeds each word as a dense vector of a preset dimensionality, the components of which are learned by maximizing the likelihood of predicting the context around it. However, as an inherent linguistic phenomenon, it is evident that there is a varying degree of difficulty in identifying words from their contexts. This suggests that a variable granularity in word vector representation may be useful to obtain sparser and more compressed word representations, requiring less storage space. To that end, in this paper, we propose a word vector training algorithm that uses a variable number of components to represent words. Given a text collection of documents, our algorithm, similar to the skip-gram approach of word2vec, learns to predict the context of a word given the current instance of a word. However, in contrast to skip-gram, which uses a static number of dimensions for each word vector, we propose to dynamically increase the dimensionality as a stochastic function of the prediction error. Our experiments with standard test collections demonstrate that our word representation method is able to achieve comparable (and sometimes even better) effectiveness than skip-gram word2vec, using a significantly smaller number of parameters (achieving compression ratio of around 65%).

Exploring Integrative Analysis Using the BioMedical Evidence Graph.

PURPOSEThe analysis of cancer biology data involves extremely heterogeneous data sets, including information from RNA sequencing, genome-wide copy number, DNA methylation data reporting on epigenetic regulation, somatic mutations from whole-exome or whole-genome analyses, pathology estimates from imaging sections or subtyping, drug response or other treatment outcomes, and various other clinical and phenotypic measurements. Bringing these different resources into a common framework, with a data model that allows for complex relationships as well as dense vectors of features, will unlock integrated data set analysis.METHODSWe introduce the BioMedical Evidence Graph (BMEG), a graph database and query engine for discovery and analysis of cancer biology. The BMEG is unique from other biologic data graphs in that sample-level molecular and clinical information is connected to reference knowledge bases. It combines gene expression and mutation data with drug-response experiments, pathway information databases, and literature-derived associations.RESULTSThe construction of the BMEG has resulted in a graph containing > 41 million vertices and 57 million edges. The BMEG system provides a graph query–based application programming interface to enable analysis, with client code available for Python, Javascript, and R, and a server online at bmeg.io. Using this system, we have demonstrated several forms of cross–data set analysis to show the utility of the system.CONCLUSIONThe BMEG is an evolving resource dedicated to enabling integrative analysis. We have demonstrated queries on the system that illustrate mutation significance analysis, drug-response machine learning, patient-level knowledge-base queries, and pathway level analysis. We have compared the resulting graph to other available integrated graph systems and demonstrated the former is unique in the scale of the graph and the type of data it makes available.

Improving spherical k-means for document clustering: Fast initialization, sparse centroid projection, and efficient cluster labeling

Due to its simplicity and intuitive interpretability, spherical k-means is often used for clustering a large number of documents. However, there exist a number of drawbacks that need to be addressed for much effective document clustering. Without well-dispersed initial points, spherical k-means fails to converge quickly, which is critical for clustering a large number of documents. Furthermore, its dense centroid vectors needlessly incorporate the impact of infrequent and less-informative words, thereby distorting the distance calculation between the document vectors.In this paper, we propose practical improvements on spherical k-means to overcome these issues during document clustering. Our proposed initialization method not only guarantees dispersed initial points, but is also up to 1000 times faster than previously well-known initialization method such as k-means++. Furthermore, we enforce sparsity on the centroid vectors by using a data-driven threshold that is capable of dynamically adjusting its value depending on the clusters. Additionally, we propose an unsupervised cluster labeling method that effectively extracts meaningful keywords to describe each cluster.We have tested our improvements on seven different text datasets that include both new and publicly available datasets. Based on our experiments on these datasets, we have found that our proposed improvements successfully overcome the drawbacks of spherical k-means in significantly reduced computation time. Furthermore, we have qualitatively verified the performance of the proposed cluster labeling method by extracting descriptive keywords of the clusters from these datasets.

Representation Learning of Knowledge Graphs With Entity Attributes

Most of the existing knowledge representation learning methods project the entities and relations represented by symbols in the knowledge graph into the low-dimensional vector space from the perspective of the structure and semantics of triples, and express the complex relations between entities and relations with dense low-dimensional vectors. However, triples in the knowledge graph not only contain relation triples, but also contain a large number of attribute triples. Existing knowledge representation methods often confuse these two kinds of triples and pay little attention to the semantic information contained in attributes and attribute values. In this paper, a novel representation learning method which makes use of the attribute information of entities is proposed. Specifically, deep convolutional neural network model is used to encode attribute information of entities, and both attribute information and triple structure information are utilized to learn knowledge representation, and then generate attribute-based representation of entities. The knowledge graph completion task was used to evaluate this method, and the experimental results on open data sets FB15K and FB24k showed that the attribute-embodied knowledge representation learning model outperforms the other baselines.

The Network Representation Learning Algorithm Based on Semi-Supervised Random Walk

As an important tool of social network analysis, network representation learning also called network embedding maps the network to a latent space and learns low-dimensional and dense real vectors of nodes, while preserving the structure and internal attributes of network. The learned representations or embedding vectors can be used for node clustering, link prediction, network visualization and other tasks for network analysis. Most of the existing network representation learning algorithms mainly focus on the preservation of micro or macro network structure, ignoring the mesoscopic community structure information. Although a few network embedding methods are proposed to preserve the community structure, they all ignore the prior information about communities. Inspired by the semi-supervised community detection in complex networks, in this article, a novel Semi-Supervised DeepWalk method(SSDW) is proposed for network representation learning, which successfully preserves the community structure of network in the embedding space. Specifically, a semi-supervised random walk sampling method which effectively integrates the pairwise constraints is proposed. By doing so, the SSDW model can guide the transition probability in the random walk process and obtain the node context sequence in line with the prior knowledge. The experimental results on eight real networks show that comparing with the popular network embedding methods, the node representation vectors integrating pairwise constraints into the random walk process can obtain higher accuracy on node clustering task, and the results of link prediction, network visualization tasks indicate that the semi-supervised model SSDW is more discriminative than unsupervised ones.

Pruning Convolutional Filters Using Batch Bridgeout

State-of-the-art computer vision models are rapidly increasing in capacity, where the number of parameters far exceeds the number required to fit the training set. This results in better optimization and generalization performance. However, the huge size of contemporary models results in large inference costs and limits their use on resource-limited devices. In order to reduce inference costs, convolutional filters in trained neural networks could be pruned to reduce the run-time memory and computational requirements during inference. However, severe post-training pruning results in degraded performance if the training algorithm results in dense weight vectors. We propose the use of Batch Bridgeout, a sparsity inducing stochastic regularization scheme, to train neural networks so that they could be pruned efficiently with minimal degradation in performance. We evaluate the proposed method on common computer vision models VGGNet, ResNet and Wide-ResNet on the CIFAR10 and CIFAR100 image classification tasks. For all the networks, experimental results show that Batch Bridgeout trained networks achieve higher accuracy across a wide range of pruning intensities compared to Dropout and weight decay regularization.

Deep Edu: A Deep Neural Collaborative Filtering for Educational Services Recommendation

In the modern world, people face an explosion of information and difficulty to find the right choice of their interest. Nowadays, people show interest in online shopping to meet their demands increasingly. For researchers and students, finding and buying the desired books from online shops is very tedious work. Recently Recommender System is an excellent tool to deal with such problems, but the Recommender System is suffering from multiple problems such as data sparsity, cold-start, and inaccuracy. To address these problems, we propose Deep Edu a novel Deep Neural Collaborative Filtering for educational services recommendation. A Deep Edu architecture consists of three parts of a Deep Neural Network model (such as input layer, a multilayered perceptron, and an output layer). The Deep Edu provides the following contributions: first, the users' identifier and books identifier features are mapped into N-dimensional dense embedding vectors, second, the Multi-Layer-Perceptron (MLP) takes the N-dimensional and non-linear features. To increase the performance of Deep Edu in all metrics, we proposed the advance Loss function. Equipped with the following, Deep Edu not only capable of learning the N-dimensional and non-linear interactions between users' identifier and books identifier, but moreover, it also considerably mitigates the cold-start, data sparsity, and inaccuracy problem. Over significant experiments performed on real-world good books dataset, the results show that Deep Edu's recommendation performance obviously outperforms existing Educational services recommendation methods.

ADPE: Adaptive Dynamic Projected Embedding on Heterogeneous Information Networks

Network embedding (NE) aims to represent network information appropriately by learning low-dimensional and dense vectors for the nodes and edges of information network. Actually, the real world is almost full of heterogeneous information networks, which stimulates the emergence of heterogeneous information networks (HINs) embedding models. However, parts of existing HIN embedding models like meta-path-based methods only capture limited and aggregated information of relations, whereas some models based on metric or distance learning are usually of high computational complexity and slow training speed. In this paper, we present a novel heterogeneous information network embedding model, which applies dynamic projection metrics and translation mechanisms to the complicated heterogeneous information networks including multiple nodes and different relations. In order to overcome the imbalance of the distribution of relations in HIN and optimize the training process, we introduce an adaptive loss function for model optimization. Further more, we propose a hybrid model with baseline method as the initialization of the model. Experiments have been implemented on some real-world HIN datasets. And empirical results show that our model significantly outperforms the state-of-the-art representation learning models.

An Approximate Model for Event Detection From Twitter Data

The abundance and real-time availability of Twitter data have proved beneficial in detecting events in various domains such as emergency situations, crime detection, public health, place recommendations, etc. Nevertheless, two critical challenges occur while detecting events using social media data. First, the uncertainty in capturing the contextual relationship among tweets, which is the result of the limited availability of the contextual information due to the small length of tweets. Second, the high computation cost required in event detection due to massive data processing. Earlier research works, addressing these challenges, have tried to capture the contextual information by using the dense vector representations of texts leveraging deep neural word embedding generation models such as Word2Vec and GloVe. However, these models are trained on the Euclidean vector space which fails to amalgamate the directional information of the vectors with the semantic information in text, incurring high computational costs. To target both the problems simultaneously, we propose modeling Twitter data as a graph-of-sentences which retains the contextual relationships while maintaining lower computational cost. The proposed model captures contextual information using JoSE, a spherical vector representation leveraging the word-word and word-paragraph semantic co-occurrence statistics in a spherical generative model. Furthermore, the framework uses the weighted-graph model to capture all the relationships among the Twitter data efficiently. The graph is further pruned with the help of the graph component filtering approach. The graph clustering model, employed to detect the events, leverages the edge weights and the partial-k clustering approach maintaining low computation costs. The experimentation on the annotated benchmark Twitter data set and the real-world datasets show improved run-time performance up to 30% while maintaining the qualitative performance (F1-score) comparable to the state-of-the-art models.

Language and Obfuscation Oblivious Source Code Authorship Attribution

Source Code Authorship Attribution can answer many interesting questions such as: Who wrote the malicious source code? Is the source code plagiarized, and does it infringe on copyright? Source Code Authorship Attribution is done by observing distinctive patterns of style in a source code whose author is unknown and comparing them with patterns learned from known authors' source codes. In this paper, we present an efficient approach to learn a novel representation using deep metric learning. The existing state of the art approaches tokenize the source code and work on the keyword level, limiting the elements of style they can consider. Our approach uses the raw character stream of source code. It can examine keywords and different stylistic features such as variable naming conventions or using tabs vs. spaces, enabling us to learn a richer representation than other keyword-based approaches. Our approach uses a character-level Convolutional Neural Network (CNN). We train the CNN to map the input character stream to a dense vector, mapping the source codes authored by the same author close to each other. In contrast, source codes written by different programmers are mapped farther apart in the embedding space. We then feed these source code vectors into the K-nearest neighbor (KNN) classifier that uses Manhattan-distance to perform authorship attribution. We validated our approach on Google Code Jam (GCJ) dataset across three different programming languages. We prepare our large-scale dataset in such a way that it does not induce type-I error. Our approach is more scalable and efficient than existing methods. We were able to achieve an accuracy of 84.94% across 20,458 authors, which is more than twice the scale of any previous study under a much more challenging setting.

Relation constrained attributed network embedding

Network embedding aims at learning a low-dimensional dense vector for each node in the network. In recent years, it has attracted great research attention due to its wide applications. Most existing studies model the graph structure only and neglect the attribute information. Although several attributed network embedding methods take the node attribute into account, they mainly focus on the basic relations between the nodes and their attributes like a user and his/her interests (attributes). The composite relations between two nodes, and two nodes’ attributes, and the related nodes and their attributes, contain rich information and can enhance the performance of many network analysis tasks. For example, two scholars having the common interests as “nature language processing” and “knowledge graph” may collaborate in the future and there will be a new edge in the network. However, such important information is still under-exploited.To address this limitation, we propose a novel framework to exploit the abundant relation information to enhance attributed network embedding. The main idea is to employ the multiple types of relations in attributed networks as the constraints to improve the network representation. To this end, we first construct the composite relations between two nodes and their attributes in addition to the commonly used basic relations. We then develop a relation constrained attributed network (RCAN) framework to learn the node representations by constraining them with these relations. We conduct extensive experiments on three real-world datasets to show the effectiveness of our proposed RCAN as an attributed network embedding method for modeling various social networks. The results demonstrate that our method achieves significantly better performance than the state-of-the-art baselines in both the link prediction and node clustering tasks.

TNAM: A tag-aware neural attention model for Top-N recommendation

Recent work shows that incorporating tag information to recommender systems is promising for improving the recommendation accuracy in social systems. However, existing approaches suffer from less reasonable assignment of tag weights when constructing the user profiles and item characteristics in real-world scenarios, resulting in decreased accuracy in making recommendations. The above issue is specifically summarized into two aspects: 1) the weight of a target item is mainly determined by number of one certain type of tags, and 2) users place equal focus on the same tag for different items. To tackle these problems, we propose a novel model named TNAM, a Tag-aware Neural Attention Model, which accurately captures users’ special attention to tags of items. In the proposed model, we design a tag-based neural attention network by extracting potential tag information to overcome the difficulty of assigning tag weights for personalized users. We combine user-item interactions with tag information to map sparse data to dense vectors in higher-order space. In this way, TNAM acquires more interrelations between users and items to make recommendations more accurate. Extensive experiments of our model on three publicly implicit feedback datasets reveal significant improvements on the metrics of HR and NDCG in Top-N recommendation tasks over several state-of-the-art approaches.

DeepEP: a deep learning framework for identifying essential proteins

BackgroundEssential proteins are crucial for cellular life and thus, identification of essential proteins is an important topic and a challenging problem for researchers. Recently lots of computational approaches have been proposed to handle this problem. However, traditional centrality methods cannot fully represent the topological features of biological networks. In addition, identifying essential proteins is an imbalanced learning problem; but few current shallow machine learning-based methods are designed to handle the imbalanced characteristics.ResultsWe develop DeepEP based on a deep learning framework that uses the node2vec technique, multi-scale convolutional neural networks and a sampling technique to identify essential proteins. In DeepEP, the node2vec technique is applied to automatically learn topological and semantic features for each protein in protein-protein interaction (PPI) network. Gene expression profiles are treated as images and multi-scale convolutional neural networks are applied to extract their patterns. In addition, DeepEP uses a sampling method to alleviate the imbalanced characteristics. The sampling method samples the same number of the majority and minority samples in a training epoch, which is not biased to any class in training process. The experimental results show that DeepEP outperforms traditional centrality methods. Moreover, DeepEP is better than shallow machine learning-based methods. Detailed analyses show that the dense vectors which are generated by node2vec technique contribute a lot to the improved performance. It is clear that the node2vec technique effectively captures the topological and semantic properties of PPI network. The sampling method also improves the performance of identifying essential proteins.ConclusionWe demonstrate that DeepEP improves the prediction performance by integrating multiple deep learning techniques and a sampling method. DeepEP is more effective than existing methods.

TEND: A Target-Dependent Representation Learning Framework for News Document

Real-time news documents published on the Internet have global financial and political impacts. Pioneering statistical approaches investigate manually defined features to capture lexical, sentiment, and event information, which suffer from feature sparsity. As a remedy, recent work has considered learning dense vector representations for documents. Such representations are general, which can not model target-dependent scenarios, such as stance detection towards a specific claim. There has been work on target-specific word and sentence representations, but little was done on target-dependent document representation. Moreover, documents contain more potentially helpful information, but also noise compared to events and sentences. To address the above issues, we focus on models that are: 1. task-driven , which optimize the neural network representations for the end task; 2. target-specific , learning news representations by considering the influence of specific targets. In particular, we propose a novel document-level target-dependent learning framework TEND. The framework employs the information of the target and the news abstract as clues, obtaining relatively informative sentences from the entire document for our objectives. The framework assembles a document representation by integrating the news abstract representation and a weighted sum of sentence representations in the document. To the best of our knowledge, we are among the first to investigate target-dependent document representation. Existing text representation models can be easily integrated into our TEND framework, and it is general enough to be applied to different target-dependent document representation tasks. We empirically evaluate our framework on two target-dependent document-level tasks, including a cumulative abnormal return prediction task and a news stance detection task. Results show that our models give the best performances compared to state-of-the-art document embedding methods, yielding robust and consistent performances across datasets.

Improving word embeddings projection for Turkish hypernym extraction

Corpus-driven approaches can automatically explore is-a relations between the word pairs from corpus. This problem is also called hypernym extraction. Formerly, lexico-syntactic patterns have been used to solve hypernym relations. The language-specific syntactic rules have been manually crafted to build the patterns. On the other hand, recent studies have applied distributional approaches to word semantics. They extracted the semantic relations relying on the idea that similar words share similar contexts. Former distributional approaches have applied one-hot bag-of-word (BOW) encoding. The dimensionality problem of BOW has been solved by various neural network approaches, which represent words in very short and dense vectors, or word embeddings. In this study, we used word embeddings representation and employed the optimized projection algorithm to solve the hypernym problem. The supervised architecture learns a mapping function so that the embeddings (or vectors) of word pairs that are in hypernym relations can be projected to each other. In the training phase, the architecture first learns the embeddings of words and the projection function from a list of word pairs. In the test phase, the projection function maps the embeddings of a given word to a point that is the closest to its hypernym. We utilized the deep learning optimization methods to optimize the model and improve the performances by tuning hyperparameters. We discussed our results by carrying out many experiments based on cross-validation. We also addressed problem-specific loss function, monitored hyperparameters, and evaluated the results with respect to different settings. Finally, we successfully showed that our approach outperformed baseline functions and other studies in the Turkish language.

Pixel-Level Remote Sensing Image Recognition Based on Bidirectional Word Vectors

In the traditional remote sensing image recognition, the traditional features (e.g., color features and texture features) cannot fully describe complex images, and the relationships between image pixels cannot be captured well. Using a single model or a traditional sequential joint model, it is easy to lose deep features during feature mining. This article proposes a new feature extraction method that uses the word embedding method from natural language processing to generate bidirectional real dense vectors to reflect the contextual relationships between the pixels. A bidirectional independent recurrent neural network (BiIndRNN) is combined with a convolutional neural network (CNN) to improve the sliced recurrent neural network (SRNN) algorithm model, which is then constructed in parallel with graph convolutional networks (GCNs) under an attention mechanism to fully exploit the deep features of images and to capture the semantic information of the context. This model is collectively named an improved SRNN and attention-treated GCN-based parallel (SAGP) model. Experiments conducted on Populus euphratica forests demonstrate that the proposed method outperforms traditional methods in terms of recognition accuracy. The validation done on public data set also proved it.

A Twelve Concentric Multilevel Twenty-Four Sided Polygonal Voltage Space Vector Structure for Variable Speed Drives

Generating dense multilevel voltage space structure using polygons of higher sides is one of the novel and elegant method to suppress low-order harmonics and to obtain refined sinusoidal voltages using voltage source inverters for variable frequency drive application without passive filters. Apart from the advantages of conventional multilevel inverter topologies, schemes generating polygonal voltage space vector structure can equip full dc-bus utilization, low switching frequency strategies to eliminate low-order harmonics, and increased linear modulation range. In this paper, an induction motor drive scheme generating a highly dense multilevel 24-sided polygonal voltage space vector structure using a single dc source, which can eliminate harmonics till 23rd order from phase voltages, is presented. Cascaded power circuit topology with a flying capacitor inverter fed with a dc source and two low voltage floating capacitor fed H-bridge inverters is used. Detailed experimentally validated results, under scalar as well as indirect rotor field oriented control of induction motor are provided. Studies on voltage ripple and reactive energy in various floating capacitors, harmonic performance of output voltage for wide range of speed operation, are also included.

Recurrent Tensor Factorization for time-aware service recommendation

With the advent of the era “everything is service ”, the emergence of Web services on the Internet is experiencing an exponential growth trend. How to recommend services to users that utilize sequential historical records has become one of the most challenging research topics in service computing. Tensor Factorization (TF) and Long Short Term Memory (LSTM) networks are two typical application paradigms for sequential service recommendation tasks. However, TF can only learn static short-term dependency patterns between users and services, ignoring the dynamic long-term dependency patterns between users and services. Although LSTM in Deep Leaning can learn dynamic long-term dependency patterns, it often encounters the trouble of vanishing gradient due to its complex gated mechanism. To address these critical challenges, we develop a novel model based on Deep Learning named Recurrent Tensor Factorization (RTF) with three innovations: (1) Three-dimensional interaction tensor of user–service-time was granulated into three fixed-size embedding dense vectors. (2) Personalized Gated Recurrent Unit (PGRU) and Generalized Tensor Factorization (GTF) simultaneously work on shared embedding dense vectors to memorize the long-term and short-term dependency patterns between users and services respectively. (3) Armed with GTF and PGRU, RTF is competent to predict the unknown Quality of Service (QoS) through comprehensive analysis. Experiments are conducted on real-world dataset, and the results indicate that our proposed method obviously outperforms six state-of-the-art time-aware service recommendation methods.

Location-Aware Deep Collaborative Filtering for Service Recommendation

With the widespread application of service-oriented architecture (SOA), a flood of similarly functioning services have been deployed online. How to recommend services to users to meet their individual needs becomes the key issue in service recommendation. In recent years, methods based on collaborative filtering (CF) have been widely proposed for service recommendation. However, traditional CF typically exploits only low-dimensional and linear interactions between users and services and is challenged by the problem of data sparsity in the real world. To address these issues, inspired by deep learning, this article proposes a new deep CF model for service recommendation, named location-aware deep CF (LDCF). This model offers the following innovations: 1) the location features are mapped into high-dimensional dense embedding vectors; 2) the multilayer-perceptron (MLP) captures the high-dimensional and nonlinear characteristics; and 3) the similarity adaptive corrector (AC) is first embedded in the output layer to correct the predictive quality of service. Equipped with these, LDCF can not only learn the high-dimensional and nonlinear interactions between users and services but also significantly alleviate the data sparsity problem. Through substantial experiments conducted on a real-world Web service dataset, results indicate that LDCF's recommendation performance obviously outperforms nine state-of-the-art service recommendation methods.