Embedding-based Methods Research Articles

Alignment-free estimation of sequence conservation for identifying functional sites using protein sequence embeddings.

Protein language modeling is a fast-emerging deep learning method in bioinformatics with diverse applications such as structure prediction and protein design. However, application toward estimating sequence conservation for functional site prediction has not been systematically explored. Here, we present a method for the alignment-free estimation of sequence conservation using sequence embeddings generated from protein language models. Comprehensive benchmarks across publicly available protein language models reveal that ESM2 models provide the best performance to computational cost ratio for conservation estimation. Applying our method to full-length protein sequences, we demonstrate that embedding-based methods are not sensitive to the order of conserved elements-conservation scores can be calculated for multidomain proteins in a single run, without the need to separate individual domains. Our method can also identify conserved functional sites within fast-evolving sequence regions (such as domain inserts), which we demonstrate through the identification of conserved phosphorylation motifs in variable insert segments in protein kinases. Overall, embedding-based conservation analysis is a broadly applicable method for identifying potential functional sites in any full-length protein sequence and estimating conservation in an alignment-free manner. To run this on your protein sequence of interest, try our scripts at https://github.com/esbgkannan/kibby.

A Cross-Modal Alignment for Zero-Shot Image Classification

Different from major classification methods based on large amounts of annotation data, we introduce a cross-modal alignment for zero-shot image classification.The key is utilizing the query of text attribute learned from the seen classes to guide local feature responses in unseen classes. First, an encoder is used to align semantic matching between visual features and their corresponding text attribute. Second, an attention module is used to get response maps through feature maps activated by the query of text attribute. Finally, the cosine distance metric is used to measure the matching degree of the text attribute and its corresponding feature response. The experiment results show that the method get better performance than existing Zero-shot Learning in embedding-based methods as well as other generative methods in CUB-200-2011 dataset.

Integrating Symbol Similarities with Knowledge Graph Embedding for Entity Alignment: An Unsupervised Framework

Entity alignment refers to discovering identical entity pairs in 2 knowledge graphs, which is a significant task in knowledge fusion. Early automated entity alignment techniques are based mainly on similarity calculation and comparing symbolic features, i.e., entity names, between entities. Nevertheless, such methods’ performance would reduce significantly when the difference between knowledge graphs is enormous because of relying on predefined comparison rules. Recently, embedding-based methods calculate the similarity between entity pairs through vector embeddings and thus can deal with different knowledge graphs. However, embedding-based methods mostly require humans to annotate data, which is laborious. Therefore, we learn from each other to propose an unsupervised entity alignment framework in this work, which can generate initial alignment seeds automatically by considering symbolic similarities. It can effectively avoid the waste of human resources and is suitable for handling multiple types of knowledge graphs. In addition, we investigate improving the quality and quantity of initial alignment by integrating multiple symbolic similarity features of entities and dealing with the situation of entity information missing better. Experimental results on 3 real datasets demonstrate its state-of-the-art performance.

A Survey of Cross-Lingual Text Classification and Its Applications on Fake News Detection

Cross-lingual text classification is a challenging task in natural language processing. The objective is to build accurate text classification models for low-resource languages by transferring the knowledge learned from high-resource languages. The task has been studied since 2003 and has attracted significantly growing attention in the last decade due to the success of deep learning models in natural language processing. Many new methods have been proposed to address the challenges in cross-lingual text classification. Meanwhile, cross-lingual fake news detection is one of the most important applications of cross-lingual text classification. It has already created significant social impacts on alleviating the infodemic problem in low-resource languages. The research works on cross-lingual text classification and cross-lingual fake news detection have been growing rapidly in recent years. Therefore, a comprehensive survey is imperative to summarize existing algorithms for cross-lingual text classification and explain the connections among them. This paper systematically reviews research works on cross-lingual text classifications and their applications in cross-lingual fake news detection. We categorize the evolution of cross-lingual text classification methods into four phases: (1) Traditional text classification models with translation; (2) Cross-lingual word embedding-based methods, (3) Pretraining then finetuning-based methods, and (4) Pretraining then prompting-based methods. We first discuss and analyze the representative methods in each phase in detail. Second, we provide a detailed review of their applications in the emerging fake news detection problem. Finally, we explore the potential issues of this open problem and also discuss possible future directions.

Study and analysis of various link predictions in knowledge graph: A challenging overview

Knowledge Graph (KG) is the network which contains some topic-based entities, called nodes, and the associated information among the entities. Here, the concept in the knowledge graph is denoted by the tuple relationship, such that the ⟨entity1, predicate, entity2⟩. The entity in the knowledge graph is the abstract concepts based on the particular object, namely the organization, dataset, people, and some associated documentation. The big issue in the KG is that it consists of some incomplete information. The missing details can be identified by employing the knowledge graph completion (KGC) solution. KGC is the same as the link prediction concepts in the knowledge graphs. However, this concept is more complex that it does not predict the link relationship among the nodes but also the diversified information from the link relations. Hence this survey analyzes different methods of link prediction techniques, and this review provides a detailed review of 50 research papers concentrating on various methods, like embedding-based methods, deep learning methods, knowledge acquisition methods, ranking methods, and representation learning methods. The analysis is carried out with respect to the survey based on the publication year, research techniques, performance measures, dataset, toolset and achievement of the research methodologies. Also, the problems in the methods are explained in the research gaps and issues. Furthermore, the future extent of these research works is done based on the limitations identified from the existing research methods.

Comprehensive Analysis of Knowledge Graph Embedding Techniques Benchmarked on Link Prediction

In knowledge graph representation learning, link prediction is among the most popular and influential tasks. Its surge in popularity has resulted in a panoply of orthogonal embedding-based methods projecting entities and relations into low-dimensional continuous vectors. To further enrich the research space, the community witnessed a prolific development of evaluation benchmarks with a variety of structures and domains. Therefore, researchers and practitioners face an unprecedented challenge in effectively identifying the best solution to their needs. To this end, we propose the most comprehensive and up-to-date study to systematically assess the effectiveness and efficiency of embedding models for knowledge graph completion. We compare 13 models on six datasets with different sizes, domains, and relational properties, covering translational, semantic matching, and neural network-based encoders. A fine-grained evaluation is conducted to compare each technique head-to-head in terms of standard metrics, training and evaluation times, memory consumption, carbon footprint, and space geometry. Our results demonstrate the high dependence between performance and graph types, identifying the best options for each scenario. Among all the encoding strategies, the new generation of translational models emerges as the most promising, bringing out the best and most consistent results across all the datasets and evaluation criteria.

Zero-Shot Embedding via Regularization-Based Recollection and Residual Familiarity Processes

The goal of zero-shot learning (ZSL) is to transfer knowledge learned from seen classes during training to unseen classes for testing, with the help of auxiliary information, such as attributes and descriptions. Most of the existing methods view ZSL as a label-embedding problem, in which class and image representations are embedded to a common space. However, many methods either show a bias toward seen classes caused by the projection domain-shift problem, or sacrifice the performance of seen classes to generalize to unseen ones. In this article, we present an embedding approach for ZSL, which is motivated by human recognition memory, namely, recollection and familiarity (R&F). We propose a decoder to regularize the nonlinear mapping between the semantic space and the visual space, which represents the reasonable recollection process, and use a residual block to refine the recognition ability for seen classes, which indicates the familiarity process. R&F can generalize well to unseen classes, while retaining the discriminative ability for the seen classes. Extensive experiments are conducted on Animals with Attribute (AwA1), Animals with Attributes 2 (AwA2), Attribute Pascal&Yahoo (aPY), SUN Attribute (SUN), Caltech-UCSD-Birds 200-2011 (CUB), and ImageNet databases. As qualitative and quantitative results show, the proposed approach outperforms state-of-the-art embedding-based methods by a large margin and significantly alleviates the projection domain-shift problem.

CAMU: Cycle-Consistent Adversarial Mapping Model for User Alignment Across Social Networks.

The user alignment problem that establishes a correspondence between users across networks is a fundamental issue in various social network analyses and applications. Since symbolic representations of users suffer from sparsity and noise when computing their cross-network similarities, the state-of-the-art methods embed users into the low-dimensional representation space, where their features are preserved and establish user correspondence based on the similarities of their low-dimensional embeddings. Many embedding-based methods try to align latent spaces of two networks by learning a mapping function before computing similarities. However, most of them learn the mapping function largely based on the limited labeled aligned user pairs and ignore the distribution discrepancy of user representations from different networks, which may lead to the overfitting problem and affect the performance. To address the above problems, we propose a cycle-consistent adversarial mapping model to establish user correspondence across social networks. The model learns mapping functions across the latent representation spaces, and the representation distribution discrepancy is addressed through the adversarial training between the mapping functions and the discriminators as well as the cycle-consistency training. Besides, the proposed model utilizes both labeled and unlabeled users in the training process, which may alleviate the overfitting problem and reduce the number of labeled users required. Results of extensive experiments demonstrate the effectiveness of the proposed model on user alignment on real social networks.

EmbedSeg: Embedding-based Instance Segmentation for Biomedical Microscopy Data.

Automatic detection and segmentation of biological objects in 2D and 3D image data is central for countless biomedical research questions to be answered. While many existing computational methods are used to reduce manual labeling time, there is still a huge demand for further quality improvements of automated solutions. In the natural image domain, spatial embedding-based instance segmentation methods are known to yield high-quality results, but their utility to biomedical data is largely unexplored. Here we introduce EmbedSeg, an embedding-based instance segmentation method designed to segment instances of desired objects visible in 2D or 3D biomedical image data. We apply our method to four 2D and seven 3D benchmark datasets, showing that we either match or outperform existing state-of-the-art methods. While the 2D datasets and three of the 3D datasets are well known, we have created the required training data for four new 3D datasets, which we make publicly available online. Next to performance, also usability is important for a method to be useful. Hence, EmbedSeg is fully open source (https://github.com/juglab/EmbedSeg), offering (i)tutorial notebooks to train EmbedSeg models and use them to segment object instances in new data, and (ii)a napari plugin that can also be used for training and segmentation without requiring any programming experience. We believe that this renders EmbedSeg accessible to virtually everyone who requires high-quality instance segmentations in 2D or 3D biomedical image data.

Dimensionality Reduction by Similarity Distance-Based Hypergraph Embedding

Dimensionality reduction (DR) is an essential pre-processing step for hyperspectral image processing and analysis. However, the complex relationship among several sample clusters, which reveals more intrinsic information about samples but cannot be reflected through a simple graph or Euclidean distance, is worth paying attention to. For this purpose, we propose a novel similarity distance-based hypergraph embedding method (SDHE) for hyperspectral images DR. Unlike conventional graph embedding-based methods that only consider the affinity between two samples, SDHE takes advantage of hypergraph embedding to describe the complex sample relationships in high order. Besides, we propose a novel similarity distance instead of Euclidean distance to measure the affinity between samples for the reason that the similarity distance not only discovers the complicated geometrical structure information but also makes use of the local distribution information. Finally, based on the similarity distance, SDHE aims to find the optimal projection that can preserve the local distribution information of sample sets in a low-dimensional subspace. The experimental results in three hyperspectral image data sets demonstrate that our SDHE acquires more efficient performance than other state-of-the-art DR methods, which improve by at least 2% on average.

HyperQuaternionE: A hyperbolic embedding model for qualitative spatial and temporal reasoning

Qualitative spatial/temporal reasoning (QSR/QTR) plays a key role in research on human cognition, e.g., as it relates to navigation, as well as in work on robotics and artificial intelligence. Although previous work has mainly focused on various spatial and temporal calculi, more recently representation learning techniques such as embedding have been applied to reasoning and inference tasks such as query answering and knowledge base completion. These subsymbolic and learnable representations are well suited for handling noise and efficiency problems that plagued prior work. However, applying embedding techniques to spatial and temporal reasoning has received little attention to date. In this paper, we explore two research questions: (1) How do embedding-based methods perform empirically compared to traditional reasoning methods on QSR/QTR problems? (2) If the embedding-based methods are better, what causes this superiority? In order to answer these questions, we first propose a hyperbolic embedding model, called HyperQuaternionE, to capture varying properties of relations (such as symmetry and anti-symmetry), to learn inversion relations and relation compositions (i.e., composition tables), and to model hierarchical structures over entities induced by transitive relations. We conduct various experiments on two synthetic datasets to demonstrate the advantages of our proposed embedding-based method against existing embedding models as well as traditional reasoners with respect to entity inference and relation inference. Additionally, our qualitative analysis reveals that our method is able to learn conceptual neighborhoods implicitly. We conclude that the success of our method is attributed to its ability to model composition tables and learn conceptual neighbors, which are among the core building blocks of QSR/QTR.

Contextualized embeddings for semantic change detection: Lessons learned

This paper presents a qualitative analysis of the outputs of contextualized embedding-based methods for detecting diachronic semantic change. We also introduce an ensemble-method outperforming previously described contextualized approaches. This method is used as a basis for an in-depth analysis of the degrees of semantic change predicted for English words across 5 decades. Our findings show that contextualized methods can sometimes predict high change scores for words which are not undergoing any real diachronic semantic shift. Such cases are discussed in detail with examples, and their linguistic categorization is proposed. Contextualized embeddings are prone to confound changes in lexicographic senses and changes in contextual variance, which naturally stems from their distributional nature. Additionally, they often merge together syntactic and semantic aspects of lexical entities. Finally, we propose possible solutions to these issues.

Graph representation learning for popularity prediction problem: A survey

The online social platforms, like Twitter, Facebook, LinkedIn and WeChat, have grown really fast in last decade and have been one of the most effective platforms for people to communicate and share information with each other. Due to the word-of-mouth effects, information usually can spread rapidly on these social media platforms. Therefore, it is important to study the mechanisms driving the information diffusion and quantify the consequence of information spread. A lot of efforts have been focused on this problem to help us better understand and achieve higher performance in viral marketing and advertising. On the other hand, the development of neural networks has blossomed in the last few years, leading to a large number of graph representation learning (GRL) models. Compared with traditional models, GRL methods are often shown to be more effective. In this paper, we present a comprehensive review for recent works leveraging GRL methods for popularity prediction problem, and categorize related literatures into two big classes, according to their mainly used model and techniques: embedding-based methods and deep learning methods. Deep learning method is further classified into convolutional neural networks, graph convolutional networks, graph attention networks, graph neural networks, recurrent neural networks, and reinforcement learning. We compare the performance of these different models and discuss their strengths and limitations. Finally, we outline the challenges and future chances for popularity prediction problem.

AEP: Aligning knowledge graphs via embedding propagation

Knowledge graph alignment aims to identify entity pairs having the same meaning between different knowledge graphs, which is essential to the automated construction of a coherent knowledge base. With the development of knowledge representation learning, researchers have proposed several useful embedding-based alignment methods. In practice, some entities are easily aligned with high confidence while others are not. However, there is a lack of effective methods which boost the alignment of implicitly aligned entities based on explicitly aligned ones. This paper presents AEP, which combines several effective schemes for accurate knowledge graph alignment. First, we employ a word-embedding model to encode the semantic information contained in entities’ surface names. We propose an attention-based graph convolutional network model to incorporate the structural information via supervised embedding propagation. Besides, we develop a multi-view bootstrapping strategy to address the over-fitting problem caused by insufficient training sets. Next, an embedding-propagation-based alignment scheme is proposed to improve the alignment accuracy by propagating explicitly aligned entities’ embeddings to implicitly aligned ones in an unsupervised manner. Finally, we conduct extensive experiments to validate the superiority of AEP and evaluate the effects of proposed schemes. Experimental results show that AEP outperforms state-of-the-art methods, and the proposed schemes improve the alignment accuracy significantly.

A Survey on Knowledge Graph-Based Recommender Systems

To solve the information explosion problem and enhance user experience in various online applications, recommender systems have been developed to model users’ preferences. Although numerous efforts have been made toward more personalized recommendations, recommender systems still suffer from several challenges, such as data sparsity and cold-start problems. In recent years, generating recommendations with the knowledge graph as side information has attracted considerable interest. Such an approach can not only alleviate the above mentioned issues for a more accurate recommendation, but also provide explanations for recommended items. In this paper, we conduct a systematical survey of knowledge graph-based recommender systems. We collect recently published papers in this field, and group them into three categories, i.e., embedding-based methods, connection-based methods, and propagation-based methods. Also, we further subdivide each category according to the characteristics of these approaches. Moreover, we investigate the proposed algorithms by focusing on how the papers utilize the knowledge graph for accurate and explainable recommendation. Finally, we propose several potential research directions in this field.

Learning representations for gene ontology terms by jointly encoding graph structure and textual node descriptors.

Measuring the semantic similarity between Gene Ontology (GO) terms is a fundamental step in numerous functional bioinformatics applications. To fully exploit the metadata of GO terms, word embedding-based methods have been proposed recently to map GO terms to low-dimensional feature vectors. However, these representation methods commonly overlook the key information hidden in the whole GO structure and the relationship between GO terms. In this paper, we propose a novel representation model for GO terms, named GT2Vec, which jointly considers the GO graph structure obtained by graph contrastive learning and the semantic description of GO terms based on BERT encoders. Our method is evaluated on a protein similarity task on a collection of benchmark datasets. The experimental results demonstrate the effectiveness of using a joint encoding graph structure and textual node descriptors to learn vector representations for GO terms.

TLogic: Temporal Logical Rules for Explainable Link Forecasting on Temporal Knowledge Graphs

Conventional static knowledge graphs model entities in relational data as nodes, connected by edges of specific relation types. However, information and knowledge evolve continuously, and temporal dynamics emerge, which are expected to influence future situations. In temporal knowledge graphs, time information is integrated into the graph by equipping each edge with a timestamp or a time range. Embedding-based methods have been introduced for link prediction on temporal knowledge graphs, but they mostly lack explainability and comprehensible reasoning chains. Particularly, they are usually not designed to deal with link forecasting -- event prediction involving future timestamps. We address the task of link forecasting on temporal knowledge graphs and introduce TLogic, an explainable framework that is based on temporal logical rules extracted via temporal random walks. We compare TLogic with state-of-the-art baselines on three benchmark datasets and show better overall performance while our method also provides explanations that preserve time consistency. Furthermore, in contrast to most state-of-the-art embedding-based methods, TLogic works well in the inductive setting where already learned rules are transferred to related datasets with a common vocabulary.

Latent Low-Rank Projection Learning with Graph Regularization for Feature Extraction of Hyperspectral Images

Due to the great benefit of rich spectral information, hyperspectral images (HSIs) have been successfully applied in many fields. However, some problems of concern also limit their further applications, such as high dimension and expensive labeling. To address these issues, an unsupervised latent low-rank projection learning with graph regularization (LatLRPL) method is presented for feature extraction and classification of HSIs in this paper, in which discriminative features can be extracted from the view of latent space by decomposing the latent low-rank matrix into two different matrices, also benefiting from the preservation of intrinsic subspace structures by the graph regularization. Different from the graph embedding-based methods that need two phases to obtain the low-dimensional projections, one step is enough for LatLRPL by constructing the integrated projection learning model, reducing the complexity and simultaneously improving the robustness. To improve the performance, a simple but effective strategy is exploited by conducting the local weighted average on the pixels in a sliding window for HSIs. Experiments on the Indian Pines and Pavia University datasets demonstrate the superiority of the proposed LatLRPL method.

Pretrained domain-specific language model for natural language processing tasks in the AEC domain

As an essential task for the architecture, engineering, and construction (AEC) industry, information processing and acquiring from unstructured textual data based on natural language processing (NLP) are gaining increasing attention. Although deep learning (DL) models for NLP tasks have been investigated for years, domain-specific pretrained DL models and their advantages are seldomly investigated in the AEC domain. Therefore, this work developed a large scale domain corpora and pretrained domain-specific language models for the AEC domain, and then systematically explores various transfer learning and fine-tuning techniques to explore the performance of pretrained DL models for various NLP tasks. First, both in-domain and close-domain Chinese corpora are developed. Then, two types of pretrained models, including static word embedding models and contextual word embedding models, are pretrained based on various domain corpora. Finally, several widely used DL models for NLP tasks are further trained and tested based on various pretrained models. The result shows that domain corpora can further improve the performance of static word embedding-based DL models and contextual word embedding-based DL models in text classification (TC) and named entity recognition (NER) tasks. Meanwhile, contextual word embedding-based DL models significantly outperform the static word embedding-based DL methods in TC and NER tasks, with maximum improvements of 8.1% and 3.8% in the F1 score, respectively. This research contributes to the body of knowledge in two ways: (1) demonstrating the advantages of domain corpora and pretrained DL models, and (2) opening the first domain-specific dataset and pretrained language models named ARCBERT for the AEC domain. Thus, this work sheds light on the adoption and application of pretrained models in the AEC domain.

Network Alignment enhanced via modeling heterogeneity of anchor nodes

Network Alignment (NA), which aims to find the nodes that represent the same entity (i.e., anchor nodes) across different networks, is a fundamental problem in many cross-network researches. Recent advances in network embedding have inspired various auspicious approaches for addressing the NA task, and embedding-based NA technology has become the main research trend. Most embedding-based NA methods follow the consistency assumption explicitly or implicitly, where anchor nodes across different networks tend to have similar local structures/neighbors. However, through the detailed statistical analysis across networks, we observe that anchor nodes have high heterogeneity, i.e., they have different local structures across different networks. Hence, in this paper, we present the formal definition of the heterogeneity of anchor nodes and propose a network alignment framework that combines heterogeneity, which can simultaneously consider the case of heterogeneity and consistency of anchor nodes. In our approach, we propose to use a variational autoencoder to learn node embeddings, and design an effective dual constraint mechanism–Laplacian regularization and heterogeneity constraint to balance the consistency and heterogeneity for network alignment across different networks respectively. Finally, to verify the effectiveness of our proposed method, we conduct extensive experiments on several real-world datasets. Experimental results show that the proposed model achieves better performance than state-of-the-art methods.