Continuous Low-dimensional Space Research Articles

RoCS: Knowledge Graph Embedding Based on Joint Cosine Similarity

Knowledge graphs usually have many missing links, and predicting the relationships between entities has become a hot research topic in recent years. Knowledge graph embedding research maps entities and relations to a low-dimensional continuous space representation to predict links between entities. The present research shows that the key to the knowledge graph embedding approach is the design of scoring functions. According to the scoring function, knowledge graph embedding methods can be classified into dot product models and distance models. We find that the triple scores obtained using the dot product model or the distance model were unbounded, which leads to large variance. In this paper, we propose RotatE Cosine Similarity (RoCS), a method to compute the joint cosine similarity of complex vectors as a scoring function to make the triple scores bounded. Our approach combines the rotational properties of the complex vector embedding model RotatE to model complex relational patterns. The experimental results demonstrate that the newly introduced RoCS yields substantial enhancements compared to RotatE across various knowledge graph benchmarks, improving up to 4.0% in hits at 1 (Hits@1) on WN18RR and improving up to 3.3% in Hits@1 on FB15K-237. Meanwhile, our method achieves some new state-of-the-art (SOTA), including Hits@3 of 95.6%, Hits@10 of 96.4% on WN18, and mean reciprocal rank (MRR) of 48.9% and Hits@1 of 44.5% on WN18RR.

Contrastive Multi-Modal Knowledge Graph Representation Learning

Representation learning of knowledge graphs (KGs) aims to embed both entities and relations as vectors in a continuous low-dimensional space, which has facilitated various applications such as link prediction and entity retrieval. Most existing KG embedding methods focus on modeling the structured fact triples independently and ignore the multi-type relations among triples as well as the variety of data types (e.g., texts and images) associated with entities in KGs, and thus fail to capture the complex and multi-modal information that is inherently inside the entity-relation triples. In this paper, we propose a novel approach for knowledge graph embedding named Contrastive Multi-modal Graph Neural Network (CMGNN), which can encapsulate comprehensive features from multi-modal content descriptions of entities and high-order connectivity structures. Specifically, CMGNN first learns entity embeddings from multi-modal content and then contrasts encodings from multi-relational local neighbors and high-order connectivities to obtain latent representations of entities and relations simultaneously. Experimental results demonstrate that CMGNN can effectively model the multi-modalities and multi-type structures in KGs, and significantly outperforms existing state-of-the-art methods on benchmark datasets for the tasks of link prediction and entity classification.

IntME: Combined Improving Feature Interactions and Matrix Multiplication for Convolution-Based Knowledge Graph Embedding

Knowledge graph embedding is a method to predict missing links in knowledge graphs by learning the interactions between embedded entities and relations in a continuous low-dimensional space. Current research on convolution-based models tends to provide sufficient interactions for extracting potential knowledge. However, sufficient interactions do not mean that they are reasonable. Our studies find that reasonable interactions can further stimulate knowledge extraction capability. Reasonable interactions need to ensure that the elements participating in interactions are disordered and in a reasonable number. To model reasonable interactions that cannot be specifically quantified, we propose a concise and effective model IntME to address this challenge. In detail, we utilize checked feature reshaping and disordered matrix multiplication to form two different types of feature maps to ensure the disorder of the interacting elements and control the number of elements before feature fusion by the shapes of the feature maps after channel scaling reshaping. In feature fusion, we employ large convolution filters and pointwise filters for the deep and shallow linear fusion of feature interactions, which can take into account both explicit and implicit knowledge extraction capability. The evaluations of four benchmark datasets show that IntME has a powerful performance in convolution-based models and a lower training cost, and also demonstrate that our proposed approaches based on reasonable interactions can effectively improve knowledge discovery capability.

Contrastive Graph Representations for Logical Formulas Embedding

Currently, the non-transparent computing process of deep learning has become a significant reason hindering its further development. The Neural-Symbolic (NS) system formed by integrating logic rules into neural networks has attracted increasing attention owing to its direct interpretability. Embedding symbolic logical formulas into a low-dimensional continuous space provides an effective way for the NS system. However, current studies are all constrained by the modeling ability for its syntactic structure and fail to preserve the intrinsic semantics in embeddings, which causes poor performance on downstream reasoning tasks. To this end, this paper proposes a novel method of <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Con</b> trastive <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">G</b> raph <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">R</b> epresentations (ConGR) for logical formulas embedding. First, to improve the modeling ability for the syntactic structure, ConGR introduces a densely connected graph convolutional network (GCN) with an attention mechanism to process syntax parsing graphs of formulas. In this way, discriminative local and global embeddings of formulas are obtained at the syntax level. Second, the contrastive instances (positive or negative) for each anchor formula are generated by the transformation under the guidance of logical properties. To preserve semantic information, two types of contrast, global-local and global-global, are carried out to refine formula embeddings. Extensive experiments demonstrate that ConGR obtains superior performance against state-of-the-art baselines on entailment checking and premise selection datasets.

Two flexible translation-based models for knowledge graph embedding

Knowledge Graph Embedding (KGE), which aims to embed the entities and relations of a knowledge gxraph into a low-dimensional continuous space, has been proven to be an effective method for completing a knowledge graph and improving the quality of the knowledge graph. The translation-based models represented by TransE, TransH, TransR and TransD have achieved great success in this regard. There is still potential for improvement in dealing with complex relations. In this paper, we find that the lack of flexibility in entity embedding limits the model’s ability to model complex relations. Therefore, we propose single-directional-flexible (sdf) models and multi-directional-flexible (mdf) models to increase the flexibility and expressiveness of entity embeddings. These two methods can be applied to the TransD model and its variant models without increasing any time cost and space cost. We conduct experiments on benchmarks such as WN18 and FB15k. The experimental results show that the models significantly surpasses the classical translation models in both tasks of triplet classification and link prediction. In particular, for Hits@1 of link prediction of WN18, we get 71.7% after applying our method to TransD, which is much better than 24.1% of TransD.

A Deep Learning-Based Sentiment Classification Model for Real Online Consumption.

Most e-commerce platforms allow consumers to post product reviews, causing more and more consumers to get into the habit of reading reviews before they buy. These online reviews serve as an emotional feedback of consumers’ product experience and contain a lot of important information, but inevitably there are malicious or irrelevant reviews. It is especially important to discover and identify the real sentiment tendency in online reviews in a timely manner. Therefore, a deep learning-based real online consumer sentiment classification model is proposed. First, the mapping relationship between online reviews of goods and sentiment features is established based on expert knowledge and using fuzzy mathematics, thus mapping the high-dimensional original text data into a continuous low-dimensional space. Secondly, after obtaining local contextual features using convolutional operations, the long-term dependencies between features are fully considered by a bidirectional long- and short-term memory network. Then, the degree of contribution of different words to the text is considered by introducing an attention mechanism, and a regular term constraint is introduced in the objective function. The experimental results show that the proposed convolutional attention–long and short-term memory network (CA–LSTM) model has a higher test accuracy of 83.3% compared with other models, indicating that the model has better classification performance.

Mixed-type data generation method based on generative adversarial networks

Data-driven based deep learing has become a key research direction in the field of artificial intelligence. Abundant training data is a guarantee for building efficient and accurate models. However, due to the privacy protection policy, research institutions are often limited to obtain a large number of training data, which would lead to a lack of training sets circumstance. In this paper, a mixed-type data generation model based on generative adversarial networks is proposed to synthesize fake data that have the same distribution with the real data, so as to supplement the real data and increase the number of available samples. The model first pre-trains the autoencoder which maps given dataset into a low-dimensional continuous space. Then, the generator constructed in the low-dimension space is obtained by training it adversarially with discriminator constructed in the original space. Since the constructed discriminator not only consider the loss of the continuous attributes but also the labeled attributes, the generator nets formed by the generator and the decoder can effectively learn the intrinsic distribution of the mixed data. We evaluate the proposed method both in the independent distribution of the attribute and in the relationship of the attributes, and the experiment results show that the proposed generate method has a better performance in preserve the intrinsic distribution compared with other generation algorithms based on deep learning.

Variational Bayesian representation learning for grocery recommendation

Representation learning has been widely applied in real-world recommendation systems to capture the features of both users and items. Existing grocery recommendation methods only represent each user and item by single deterministic points in a low-dimensional continuous space, which limit the expressive ability of their embeddings, resulting in recommendation performance bottlenecks. In addition, existing representation learning methods for grocery recommendation only consider the items (products) as independent entities, neglecting their other valuable side information, such as the textual descriptions and the categorical data of items. In this paper, we propose the Variational Bayesian Context-Aware Representation (VBCAR) model for grocery recommendation. VBCAR is a novel variational Bayesian model that learns distributional representations of users and items by leveraging basket context information from historical interactions. Our VBCAR model is also extendable to leverage side information by encoding contextual features into representations based on the inference encoder. We conduct extensive experiments on three real-world grocery datasets to assess the effectiveness of our model as well as the impact of different construction strategies for item side information. Our results show that our VBCAR model outperforms the current state-of-the-art grocery recommendation models while integrating item side information (especially the categorical features with the textual information of items) results in further significant performance gains. Furthermore, we demonstrate through analysis that our model is able to effectively encode similarities between product types, which we argue is the primary reason for the observed effectiveness gains.

Location-Sensitive Embedding for Knowledge Graph Embedding

<p indent=0mm>Knowledge graph embedding maps the discrete symbolic entities and relations into continuous low-dimensional space, enabling the downstream task (e.g., inference, knowledge graph completion and etc.) to be carried on in an algebraic manner. Roughly speaking, there are two main-streams towards knowledge graph embedding, translational distance models and semantic matching models. The learning bottleneck of the former caused by special graph structures is analyzed. And it is found out that translational distance models are to be blamed for treating head and tail entities from a same viewpoint. So, a location-sensitive embedding (LSE) model is proposed. Unlike previous models, it just transforms the head entity with relational-specific mapping. And it models relation as a general linear transformation instead of a translational operator. Its representation capacity and the relationship to current models are theoretically analyzed. Also, to make it efficient, the model is simplified by assuming a diagonal matrix for transformation, forming a simplified model LSE<italic>_d</italic>. It is evaluated that the proposed model on four large-scale knowledge graph datasets for link prediction task. Experiments have shown that the proposed model achieves highest, or at least competitive, performance compared with the state-of-the-art models.

Knowledge graph embedding with shared latent semantic units

Knowledge graph embedding (KGE) aims to project both entities and relations into a continuous low-dimensional space. However, for a given knowledge graph (KG), only a small number of entities and relations occur many times, while the vast majority of entities and relations occur less frequently. This data sparsity problem has largely been ignored by most of the existing KGE models. To this end, in this paper, we propose a general technique to enable knowledge transfer among semantically similar entities or relations. Specifically, we define latent semantic units (LSUs), which are the sub-components of entity and relation embeddings. Semantically similar entities or relations are supposed to share the same LSUs, and thus knowledge can be transferred among entities or relations. Finally, extensive experiments show that the proposed technique is able to enhance existing KGE models and can provide better representations of KGs.

Spatial-temporal attention network for multistep-ahead forecasting of chlorophyll

The multistep-ahead prediction of chlorophyll provides an effective means for early warning of red tide. However, since multistep-ahead forecasting presents challenges, such as vague interactive relationships among ocean factors, long-term dependence modeling, and accumulative errors, existing methods mostly concentrate on the current time or one-step-ahead forecasting. In this paper, a hierarchical multistep-ahead forecasting model spatial-temporal attention network(STAN), which integrates the spatial context extractor network(SCE-net), long short-term memory network(LSTM), and the temporal attention mechanism, is proposed for the prediction of chlorophyll. In STAN, the input layer utilizes SCE-net to excavate relationships among ocean factors and generate high-level semantic via embedding factors into a continuous low-dimensional space. The middle layer applies LSTM to build the long-term dependencies of corresponding semantic representations. The output layer uses another LSTM with temporal attention to reduce accumulative errors and maintain temporal continuity. The attention can assign different weights to the middle layer’s hidden state and generate a context vector. Then the context vector and the final predicted value are considered as the current input for better forecasting. The buoy observation data of the Xiamen coastal area monitored in 2009–2011 is used to verify the efficiency of STAN. Experimental results prove that STAN outperforms the state-of-the-art methods of multistep-ahead prediction. When using 7 observation steps to forecast 15 steps, the MAPE of STAN is 0.3209, and the MAE is 0.1 lower than the values of the baselines approaches.

Utilizing Textual Information in Knowledge Graph Embedding: A Survey of Methods and Applications

Techniques that map the entities and relations of the knowledge graph (KG) into a low-dimensional continuous space are called KG embedding or knowledge representation learning. However, most existing techniques learn the embeddings based on the facts in KG alone, suffering from the issues of imperfection and spareness of KG. Recently, the research on textual information in KG embedding has attracted much attention due to the rich semantic information supplied by the texts. Thus, in this paper, a survey of techniques for textual information based KG embedding is proposed. Firstly, we introduce the techniques for encoding the textual information to represent the entities and relations from perspectives of encoding models and scoring functions, respectively. Secondly, methods for incorporating the textual information in the existing embedding techniques are summarized. Thirdly, we discuss the training procedure of textual information based KG embedding techniques. Finally, applications of KG embedding with textual information in the specific tasks such as KG completion in zero-shot scenario, multilingual entity alignment, relation extraction and recommender system are explored. We hope that this survey will give insights to researchers into textual information based KG embedding.

Deciphering protein evolution and fitness landscapes with latent space models

Protein sequences contain rich information about protein evolution, fitness landscapes, and stability. Here we investigate how latent space models trained using variational auto-encoders can infer these properties from sequences. Using both simulated and real sequences, we show that the low dimensional latent space representation of sequences, calculated using the encoder model, captures both evolutionary and ancestral relationships between sequences. Together with experimental fitness data and Gaussian process regression, the latent space representation also enables learning the protein fitness landscape in a continuous low dimensional space. Moreover, the model is also useful in predicting protein mutational stability landscapes and quantifying the importance of stability in shaping protein evolution. Overall, we illustrate that the latent space models learned using variational auto-encoders provide a mechanism for exploration of the rich data contained in protein sequences regarding evolution, fitness and stability and hence are well-suited to help guide protein engineering efforts.

TDN: An Integrated Representation Learning Model of Knowledge Graphs

Knowledge graph (KG) is playing an important role in many artificial intelligence applications. Representation learning of KGs aims to project both entities and relations into a continuous low-dimensional space. The representation learning technique based on embedding has been used to implement the KG completion, which aims to predict potential triples (head, relation, and tail) in KG. Most current methods concentrate on learning representations based on triple information while ignoring integrating the textual knowledge and network topology of KG. This leads to ambiguous completions. To address this problem and implement more accurate KG completion, we propose a new representation learning model, TDN model, which integratedly embeds the information of triples, text descriptions, and network structure of KG in a low-dimensional vector space. The framework of TDN is defined and the methodology of implementing TDN embedding is explored. To verify the effectiveness of the proposed model, we evaluate TDN via the experiments of link prediction on the real-world datasets. The experimental results confirm the above claims and show that TDN-based embedding significantly outperforms other baselines.

Distributed representation learning for knowledge graphs with entity descriptions

Recent studies of knowledge representation attempt to project both entities and relations, which originally compose a high-dimensional and sparse knowledge graph, into a continuous low-dimensional space. One canonical approach TransE [2] which represents entities and relations with vectors (embeddings), achieves leading performances solely with triplets, i.e. (head_entity, relation, tail_entity), in a knowledge base. The cutting-edge method DKRL [23] extends TransE via enhancing the embeddings with entity descriptions by means of deep neural network models. However, DKRL requires extra space to store parameters of inner layers, and relies on more hyperparameters to be tuned. Therefore, we create a single-layer model which requests much fewer parameters. The model measures the probability of each triplet along with corresponding entity descriptions, and learns contextual embeddings of entities, relations and words in descriptions simultaneously, via maximizing the loglikelihood of the observed knowledge. We evaluate our model in the tasks of knowledge graph completion and entity type classification with two benchmark datasets: FB500K and EN15K, respectively. Experimental results demonstrate that the proposed model outperforms both TransE and DKRL, indicating that it is both efficient and effective in learning better distributed representations for knowledge bases.

Representation Learning of Knowledge Graphs with Entity Descriptions

Representation learning (RL) of knowledge graphs aims to project both entities and relations into a continuous low-dimensional space. Most methods concentrate on learning representations with knowledge triples indicating relations between entities. In fact, in most knowledge graphs there are usually concise descriptions for entities, which cannot be well utilized by existing methods. In this paper, we propose a novel RL method for knowledge graphs taking advantages of entity descriptions. More specifically, we explore two encoders, including continuous bag-of-words and deep convolutional neural models to encode semantics of entity descriptions. We further learn knowledge representations with both triples and descriptions. We evaluate our method on two tasks, including knowledge graph completion and entity classification. Experimental results on real-world datasets show that, our method outperforms other baselines on the two tasks, especially under the zero-shot setting, which indicates that our method is capable of building representations for novel entities according to their descriptions. The source code of this paper can be obtained from https://github.com/xrb92/DKRL.

VISUALIZING RELATIONS USING THE "OBSERVABLE REPRESENTATION"

The observable representation provides an embedding of a discrete space in a low dimensional continuous space. Typically, the discrete space is a model of a complex system. This graphical representation is known to highlight significant properties of the original space and can serendipitously reveal unanticipated relationships. We report on the current status of this technique and give examples of its applications and rationale.

Mean-Field Spin Glass in the Observable Representation

The state space for the N-spin mean-field (Sherrington-Kirkpatrick) spin glass - nominally an N-cube - is embedded in a low dimensional continuous space in such a way that metastable and stable phases can easily be discerned, a concept of nearness of configurations defined, and peaks in the Parisi q-parameter overlap distribution identified. The dynamical and partly hierarchical interrelation of these phases can be directly imaged.