Existing Embedding Approaches Research Articles

Attention‐based network embedding with higher‐order weights and node attributes

AbstractNetwork embedding aspires to learn a low‐dimensional vector of each node in networks, which can apply to diverse data mining tasks. In real‐life, many networks include rich attributes and temporal information. However, most existing embedding approaches ignore either temporal information or network attributes. A self‐attention based architecture using higher‐order weights and node attributes for both static and temporal attributed network embedding is presented in this article. A random walk sampling algorithm based on higher‐order weights and node attributes to capture network topological features is presented. For static attributed networks, the algorithm incorporates first‐order to k‐order weights, and node attribute similarities into one weighted graph to preserve topological features of networks. For temporal attribute networks, the algorithm incorporates previous snapshots of networks containing first‐order to k‐order weights, and nodes attribute similarities into one weighted graph. In addition, the algorithm utilises a damping factor to ensure that the more recent snapshots allocate a greater weight. Attribute features are then incorporated into topological features. Next, the authors adopt the most advanced architecture, Self‐Attention Networks, to learn node representations. Experimental results on node classification of static attributed networks and link prediction of temporal attributed networks reveal that our proposed approach is competitive against diverse state‐of‐the‐art baseline approaches.

Towards Understanding the Instability of Network Embedding

Network embedding algorithms learn a mapping from the discrete representation of nodes to continuous vector spaces that preserve the proximities of nodes. The techniques have a wide range of applications in various downstream tasks such as node classification, link prediction, and network alignment. Despite recent efforts to the design of novel models, little attention has been paid to understanding the instability of network embedding. In this paper, we fill this gap by investigating several state-of-the-art network embedding methods. Node embeddings form a geometric shape in the latent space. Characterizing the geometry is critical to figure out the variance of network embedding. Hence, we define two metrics to characterize the geometric properties and find that node embeddings tremble in different instantiations of an embedding space. Then, we formally define the stability of node embeddings as the invariance of the nearest neighbors of nodes. Experimental results show that existing embedding approaches have significant amounts of instability. We explore the influence factors that affect the stability of different methods and find that both network structure and algorithm models affect the stability of node embeddings significantly. Finally, we examine the implications of embedding instability for downstream tasks and find remarkable impacts on the performance.

Using molecular embeddings in QSAR modeling: does it make a difference?

With the consolidation of deep learning in drug discovery, several novel algorithms for learning molecular representations have been proposed. Despite the interest of the community in developing new methods for learning molecular embeddings and their theoretical benefits, comparing molecular embeddings with each other and with traditional representations is not straightforward, which in turn hinders the process of choosing a suitable representation for Quantitative Structure-Activity Relationship (QSAR) modeling. A reason behind this issue is the difficulty of conducting a fair and thorough comparison of the different existing embedding approaches, which requires numerous experiments on various datasets and training scenarios. To close this gap, we reviewed the literature on methods for molecular embeddings and reproduced three unsupervised and two supervised molecular embedding techniques recently proposed in the literature. We compared these five methods concerning their performance in QSAR scenarios using different classification and regression datasets. We also compared these representations to traditional molecular representations, namely molecular descriptors and fingerprints. As opposed to the expected outcome, our experimental setup consisting of over $25 000$ trained models and statistical tests revealed that the predictive performance using molecular embeddings did not significantly surpass that of traditional representations. Although supervised embeddings yielded competitive results compared with those using traditional molecular representations, unsupervised embeddings tended to perform worse than traditional representations. Our results highlight the need for conducting a careful comparison and analysis of the different embedding techniques prior to using them in drug design tasks and motivate a discussion about the potential of molecular embeddings in computer-aided drug design.

Pattern2Vec: Representation of clickstream data sequences for learning user navigational behavior

SummaryWord embedding approaches represent data sequences to handle their contextual meaning in the NLP tasks. Nowadays, there is an emerging need to understand the user behavior patterns over navigational clickstream data. However, representing the URL data sequences utilizing existing embedding approaches to cluster users' behavior with unsupervised machine learning tasks is a challenging task. This study introduces the Patter2Vec embedding approach using a representation vector to construct contextual, precise, and interpretable clusters over the hidden and popular navigational patterns. To test the usability of the proposed representation in clustering tasks, we conduct an experimental study, which indicates that Pattern2Vec outperforms existing embedding approaches.

Geometricus represents protein structures as shape-mers derived from moment invariants.

As the number of experimentally solved protein structures rises, it becomes increasingly appealing to use structural information for predictive tasks involving proteins. Due to the large variation in protein sizes, folds and topologies, an attractive approach is to embed protein structures into fixed-length vectors, which can be used in machine learning algorithms aimed at predicting and understanding functional and physical properties. Many existing embedding approaches are alignment based, which is both time-consuming and ineffective for distantly related proteins. On the other hand, library- or model-based approaches depend on a small library of fragments or require the use of a trained model, both of which may not generalize well. We present Geometricus, a novel and universally applicable approach to embedding proteins in a fixed-dimensional space. The approach is fast, accurate, and interpretable. Geometricus uses a set of 3D moment invariants to discretize fragments of protein structures into shape-mers, which are then counted to describe the full structure as a vector of counts. We demonstrate the applicability of this approach in various tasks, ranging from fast structure similarity search, unsupervised clustering and structure classification across proteins from different superfamilies as well as within the same family. Python code available at https://git.wur.nl/durai001/geometricus.

PixelSNE: Pixel‐Aligned Stochastic Neighbor Embedding for Efficient 2D Visualization with Screen‐Resolution Precision

AbstractEmbedding and visualizing large‐scale high‐dimensional data in a two‐dimensional space is an important problem, because such visualization can reveal deep insights of complex data. However, most of the existing embedding approaches run on an excessively high precision, even when users want to obtain a brief insight from a visualization of large‐scale datasets, ignoring the fact that in the end, the outputs are embedded onto a fixed‐range pixel‐based screen space. Motivated by this observation and directly considering the properties of screen space in an embedding algorithm, we propose Pixel‐Aligned Stochastic Neighbor Embedding (PixelSNE), a highly efficient screen resolution‐driven 2D embedding method which accelerates Barnes‐Hut tree‐based t‐distributed stochastic neighbor embedding (BH‐SNE), which is known to be a state‐of‐the‐art 2D embedding method. Our experimental results show a significantly faster running time for PixelSNE compared to BH‐SNE for various datasets while maintaining comparable embedding quality.

VINEA: An Architecture for Virtual Network Embedding Policy Programmability

Network virtualization has enabled new business models by allowing infrastructure providers to lease or share their physical network. A fundamental management problem that cloud providers face to support customized virtual network (VN) services is the virtual network embedding. This requires solving the (NP-hard) problem of matching constrained virtual networks onto the physical network. In this paper we present VINEA, a policy-based virtual network embedding architecture, and its system implementation. VINEA leverages our previous results on VN embedding optimality and convergence guarantees, and it is based on a network utility maximization approach that separates policies (i.e., high-level goals) from underlying embedding mechanisms: resource discovery, virtual network mapping, and allocation on the physical infrastructure. We show how VINEA can subsume existing embedding approaches, and how it can be used to design novel solutions that adapt to different scenarios, by merely instantiating different policies. We describe the VINEA architecture, as well as our object model: our VINO protocol and the API to program the embedding policies; we then analyze key representative tradeoffs among novel and existing VN embedding policy configurations, via event-driven simulations, and with our prototype implementation. Among our findings, our evaluation shows how, in contrast to existing solutions, simultaneously embedding nodes and links may lead to lower providers’ revenue. We release our implementation on a testbed that uses a Linux system architecture to reserve virtual node and link capacities. Our prototype can be also used to augment existing open-source “Networking as a Service” architectures such as OpenStack Neutron, that currently lacks a VN embedding protocol, and as a policy-programmable solution to the “slice stitching” problem within wide-area virtual network testbeds.

Manifold regularized cross-modal embedding for zero-shot learning

Zero-Shot Learning (ZSL) aims at classifying previously unseen class samples and has gained its popularity in applications where samples of some categories are scarce for training. The basic idea to address this issue is transferring knowledge from the seen classes to the unseen classes through mapping the visual feature to an embedding space spanned by class semantic information. The class semantic information can be obtained from human-labeled attributes or text corpus in an unsupervised fashion. Therefore, the embedding function from visual space to the embedding space is extremely important. However, the existing embedding approaches to ZSL mainly focus on aligning pairwise semantic consistency from heterogeneous spaces but ignore the intrinsic structure of the locally homogeneous isomorph. In order to preserve the locally visual structure in the embedding process, this paper proposes a Manifold regularized Cross-Modal Embedding (MCME) approach for ZSL by formulating the manifold constraint for intrinsic structure of the visual features as well as aligning pairwise consistency. The linear, closed-form solution makes MCME efficient to compute. Furthermore, rather than applying the embedding function learned from the seen classes directly, we also propose a new domain adaptation strategy to overcome the domain-shift problem during the knowledge transfer process. The MCME with the domain adaptation method is called MCME-DA. Extensive experiments on the benchmark datasets of AwA and CUB validate the superiority and promise of MCME and MCME-DA.