Existing Embedding Models Research Articles

An Adaptive Semantic Mining Framework for Heterogeneous Information Network Embedding

Heterogeneous information network (HIN) embedding aims to map heterogeneous nodes to the low-dimensional vector space. The existing embedding models cannot determine the optimal length of semantics automatically and reveal full semantic information adaptively for different heterogeneous networks. To address this challenge, an HIN embedding model with adaptive semantic mining is proposed. First, we project heterogeneous nodes into the same space and aggregate the features of target types in the first-degree range. Then, the semantics of different node types is combined through the attention mechanism, and latent meta-paths are mined using the attention coefficients. Finally, multiple feature aggregation layers are stacked with residual blocks. The residual weights control the proportion of semantics transferred between layers to aggregate more distal features selectively. In addition, we designed the Selected DropLink unit to remove links which transfer negative information, which can further improve the resistance of model to over-smoothing. Experiments show that our model can obtain more accurate embedding results and can automatically mine complex semantic connections between heterogeneous nodes without prior definition of meta-path and semantic depth.

Multi-view learning-based heterogeneous network representation learning

Network representation learning is an important tool for extracting latent features from heterogeneous networks to enhance downstream analysis tasks. However, for heterogeneous networks in the era of big data, their heterogeneity, unseen network noises, latent structural and semantic features, and the coupling degree between the two features are increasing. These unignorable changes lead to the difficulties of extracting latent features and the demand for further optimization of existing embedding models. In this paper, an unsupervised multi-view learning based heterogeneous network representation model is proposed, called MVHNR. Firstly, to simplify the difficulty of feature learning, a novel view generation strategy is designed to divide the heterogeneous network into multiple sub-views that contain only one semantic feature and its related structural features. Then, to improve the accuracy and robustness of feature learning with network noises, a novel adjacency matrix strategy is constructed for each view, and a novel adversarial learning strategy is used to both learn the local semantic and structural features in each view. Finally, to aggregate the local features of all views, a multi-headed attention network is used to generate a global feature vector of each node. Extensive experimental results on three tasks and five public networks show the advantages of our MVHNR model.

A Knowledge Graph Embedding Framework With Triple Semantics

The knowledge graph embedding model aims to use low-dimensional real-valued vectors to represent the entities and relations in the triples, where operations such as link prediction and triple classification can be performed based on these representations. However, existing embedding models only consider the structural embedding of triples, while ignoring the semantic information of triples. This paper proposes a knowledge graph embedding learning framework combined with triple semantic information (KGSE). KGSE comprehensively considers the structural embedding and semantic embedding of triples, where semantic embedding is used as a supplement to improve the quality of embedding. Specifically, KGSE uses the improved TransD model to obtain the structural embedding of triples, and employs the deep convolutional neural model combined with an attention mechanism to obtain the semantic embedding of triples. In addition, a novel energy function is designed to jointly train the above two embeddings. Experimental results show that the proposed framework improves significantly compared with Trans-based models and other baseline models in link prediction and triple classification tasks, which verifies the effectiveness of the proposed framework.

HyperQuaternionE: A hyperbolic embedding model for qualitative spatial and temporal reasoning

Qualitative spatial/temporal reasoning (QSR/QTR) plays a key role in research on human cognition, e.g., as it relates to navigation, as well as in work on robotics and artificial intelligence. Although previous work has mainly focused on various spatial and temporal calculi, more recently representation learning techniques such as embedding have been applied to reasoning and inference tasks such as query answering and knowledge base completion. These subsymbolic and learnable representations are well suited for handling noise and efficiency problems that plagued prior work. However, applying embedding techniques to spatial and temporal reasoning has received little attention to date. In this paper, we explore two research questions: (1) How do embedding-based methods perform empirically compared to traditional reasoning methods on QSR/QTR problems? (2) If the embedding-based methods are better, what causes this superiority? In order to answer these questions, we first propose a hyperbolic embedding model, called HyperQuaternionE, to capture varying properties of relations (such as symmetry and anti-symmetry), to learn inversion relations and relation compositions (i.e., composition tables), and to model hierarchical structures over entities induced by transitive relations. We conduct various experiments on two synthetic datasets to demonstrate the advantages of our proposed embedding-based method against existing embedding models as well as traditional reasoners with respect to entity inference and relation inference. Additionally, our qualitative analysis reveals that our method is able to learn conceptual neighborhoods implicitly. We conclude that the success of our method is attributed to its ability to model composition tables and learn conceptual neighbors, which are among the core building blocks of QSR/QTR.

Temporal Knowledge Graph Completion Using Box Embeddings

Knowledge graph completion is the task of inferring missing facts based on existing data in a knowledge graph. Temporal knowledge graph completion (TKGC) is an extension of this task to temporal knowledge graphs, where each fact is additionally associated with a time stamp. Current approaches for TKGC primarily build on existing embedding models which are developed for static knowledge graph completion, and extend these models to incorporate time, where the idea is to learn latent representations for entities, relations, and timestamps and then use the learned representations to predict missing facts at various time steps. In this paper, we propose BoxTE, a box embedding model for TKGC, building on the static knowledge graph embedding model BoxE. We show that BoxTE is fully expressive, and possesses strong inductive capacity in the temporal setting. We then empirically evaluate our model and show that it achieves state-of-the-art results on several TKGC benchmarks

Word Embedding Quantization for Personalized Recommendation on Storage-Constrained Edge Devices in a Smart Store

In recent years, word embedding models receive tremendous research attentions due to their capability of capturing textual semantics. This study investigates the issue of employing word embedding models into storage-constrained edge devices for personalized item-of-interest recommendation in a smart store. The challenge lies in that the existing embedding models are often too large to fit into a storage-constrained edge device. One naive idea is to reside the word embedding model in a secondary storage and process recommendation with that storage. However, this idea suffers from the burden of additional traffics. To this end, we propose a framework called Word Embedding Quantization (WEQ) which constructs an index upon a given word embedding model and stores the index in the primary storage to enable the use of the word embedding model by edge devices. One challenge for using the index is that the exact user profile is no longer ensured. However, we find that there are opportunities for computing the correct recommendation results by knowing only an inexact user profile. In this paper, we propose a series of techniques that leverage the opportunities for computing candidates with the goal of minimizing the accessing cost to a secondary storage in edge devices. Experiments are performed to verify the efficiency of the proposed techniques, demonstrating the feasibility of the proposed framework.

Binarized Embeddings for Fast, Space-Efficient Knowledge Graph Completion

Methods based on vector embeddings of knowledge graphs have been actively pursued as a promising approach to knowledge graph completion. However, existing embedding models generate storage-inefficient representations, particularly when the number of entities and relations, and the dimensionality of the real-valued embedding vectors are large. We present a binarized CANDECOMP/PARAFAC (CP) decomposition algorithm, which we refer to as B-CP, where real-valued parameters are replaced by binary values to reduce model size. Moreover, a fast score computation technique is developed with bitwise operations. We prove that B-CP is fully expressive given a sufficiently large dimentionality of embedding vectors. Experimental results on several benchmark datasets demonstrate that the proposed method successfully reduces model size by more than an order of magnitude while maintaining task performance at the same level as the real-valued CP model.

Towards Fine-Grained Temporal Network Representation via Time-Reinforced Random Walk

Encoding a large-scale network into a low-dimensional space is a fundamental step for various network analytic problems, such as node classification, link prediction, community detection, etc. Existing methods focus on learning the network representation from either the static graphs or time-aggregated graphs (e.g., time-evolving graphs). However, many real systems are not static or time-aggregated as the nodes and edges are timestamped and dynamically changing over time. For examples, in anti-money laundering analysis, cycles formed with time-ordered transactions might be red flags in online transaction networks; in novelty detection, a star-shaped structure appearing in a short burst might be an underlying hot topic in social networks. Existing embedding models might not be able to well preserve such fine-grained network dynamics due to the incapability of dealing with continuous-time and the negligence of fine-grained interactions. To bridge this gap, in this paper, we propose a fine-grained temporal network embedding framework named FiGTNE, which aims to learn a comprehensive network representation that preserves the rich and complex network context in the temporal network. In particular, we start from the notion of fine-grained temporal networks, where the temporal network can be represented as a series of timestamped nodes and edges. Then, we propose the time-reinforced random walk (TRRW) with a bi-level context sampling strategy to explore the essential structures and temporal contexts in temporal networks. Extensive experimental results on real graphs demonstrate the efficacy of our FiGTNE framework.

LENA: Locality-Expanded Neural Embedding for Knowledge Base Completion

Embedding based models for knowledge base completion have demonstrated great successes and attracted significant research interest. In this work, we observe that existing embedding models all have their loss functions decomposed into atomic loss functions, each on a triple or an postulated edge in the knowledge graph. Such an approach essentially implies that conditioned on the embeddings of the triple, whether the triple is factual is independent of the structure of the knowledge graph. Although arguably the embeddings of the entities and relation in the triple contain certain structural information of the knowledge base, we believe that the global information contained in the embeddings of the triple can be insufficient and such an assumption is overly optimistic in heterogeneous knowledge bases. Motivated by this understanding, in this work we propose a new embedding model in which we discard the assumption that the embeddings of the entities and relation in a triple is a sufficient statistic for the triple’s factual existence. More specifically, the proposed model assumes that whether a triple is factual depends not only on the embedding of the triple but also on the embeddings of the entities and relations in a larger graph neighbourhood. In this model, attention mechanisms are constructed to select the relevant information in the graph neighbourhood so that irrelevant signals in the neighbourhood are suppressed. Termed locality-expanded neural embedding with attention (LENA), this model is tested on four standard datasets and compared with several stateof-the-art models for knowledge base completion. Extensive experiments suggest that LENA outperforms the existing models in virtually every metric.

A distant supervision method based on paradigmatic relations for learning word embeddings

Word embeddings learned on external resources have succeeded in improving many NLP tasks. However, existing embedding models still face challenges in situations where fine-gained semantic information is required, e.g., distinguishing antonyms from synonyms. In this paper, a distant supervision method is proposed to guide the training process by introducing semantic knowledge in a thesaurus. Specifically, the proposed model shortens the distance between target word and its synonyms by controlling the movements of them in both unidirectional and bidirectional, yielding three different models, namely Unidirectional Movement of Target Model (UMT), Unidirectional Movement of Synonyms Model (UMS) and Bidirectional Movement of Target and Synonyms Model (BMTS). Extensive computational experiments have been conducted, and results are collected for analysis purpose. The results show that the proposed models not only efficiently capture semantic information of antonyms but also achieve significant improvements in both intrinsic and extrinsic evaluation tasks. To validate the performance of the proposed models (UMT, UMS and BMTS), results are compared against well-known models, namely Skip-gram, JointRCM, WE-TD and dict2vec. The performances of the proposed models are evaluated on four tasks (benchmarks): word analogy (intrinsic), synonym-antonym detection (intrinsic), sentence matching (extrinsic) and text classification (extrinsic). A case study is provided to illustrate the working of the proposed models in an effective manner. Overall, a distant supervision method based on paradigmatic relations is proposed for learning word embeddings and it outperformed when compared against other existing models.

Improving Recommendations by Embedding Multi-Entity Relationships With Latent Dual-Metric Learning

Recently, latent vector embedding has become a research hotspot, with its great representative ability to measure the latent relationships among different views. However, most researches utilize the inner product of latent vectors as the representation of relationships, and they develop some embedding models based on this theory. In this paper, we take deep insight into the existing embedding models and find that utilizing the inner product may increase several problems: 1) in latent space, the inner product among three vectors may violate triangle principle; 2) the inner product cannot measure the relationships between vectors in the same category, such as user and user and item and item; and 3) the inner product cannot catch the collaborative relationships (user-user and item-item) for collaborative filtering. Along with this line, we propose a latent vector embedding model for collaborative filtering: latent dual metric embedding (LDME), which utilizes the dual-Euclidean distance in latent space, instead of the inner product, to represent different types of relationships (user-user, item-item, and user-item) with a uniform framework. Specifically, we design an embedding loss function in LDME, which can measure the close and remote relationships between entities, tackle the above problems, and achieve a more clear, well-explained embedding result. Extensive experiments are conducted on several real-world datasets (Amazon, Yelp, Taobao, and Jingdong), where the expiring results demonstrate that LDME can overperform some state-of-the-art user-item embedding models and can benefit the existing collaborative filtering models.