Embedding Framework Research Articles

FedGE: Break the Scalability Limitation of Graph Neural Network With Federated Graph Embedding

Neighborhood aggregation algorithms, represented as graph convolutional networks, have attained non-negligible success in numerous topological structure-based scenarios with the assumption that the topological structure of the given graph is pre-defined and relatively small. However, a real-world graph generally is a super large graph consisting of many small graphs that are interconnected and overlapping. Which makes graph embedding in real-life industries, by nature, fall into the federated learning scheme. While current graph-based algorithms are only able to capture the individual topology of each natural graph, learning the complete structural information of the merged large graph remains challenging due to the unsustainable computational cost of graph convolutional operations. We propose a tailored federated graph embedding framework to learn the intact structural information of the numerous inherently linked small-scale graphs and the embedding of each node. We leverage graphs with around two and a half million nodes to validate the effectiveness and the correctness of the proposed framework.

Universal Knowledge Graph Embedding Framework Based on High-Quality Negative Sampling and Weighting

The traditional model training approach based on negative sampling randomly samples a portion of negative samples for training, which can easily overlook important negative samples and adversely affect the training of knowledge graph embedding models. Some researchers have explored non-sampling model training frameworks that use all unobserved triples as negative samples to improve model training performance. However, both training methods inevitably introduce false negative samples and easy-to-separate negative samples that are far from the model’s decision boundary, and they do not consider the adverse effects of long-tail entities and relations during training, thus limiting the improvement of model training performance. To address this issue, we propose a universal knowledge graph embedding framework based on high-quality negative sampling and weighting, called HNSW-KGE. First, we conduct pre-training based on the NS-KGE non-sampling training framework to quickly obtain an initial set of relatively high-quality embedding vector representations for all entities and relations. Second, we design a candidate negative sample set construction strategy that samples a certain number of negative samples that are neither false negatives nor easy-to-separate negatives for all positive triples, based on the embedding vectors obtained from pre-training. This ensures the provision of high-quality negative samples for model training. Finally, we apply weighting to the loss function based on the frequency of the entities and relations appearing in the triples to mitigate the adverse effects of long-tail entities and relations on model training. Experiments conducted on benchmark datasets FB15K237 and WN18RR using various knowledge graph embedding models demonstrate that our proposed framework HNSW-KGE, based on high-quality negative sampling and weighting, achieves better training performance and exhibits versatility, making it applicable to various types of knowledge embedding models.

Enhanced Machine Learning Framework for Autonomous Depression Detection Using Modwave Cepstral Fusion and Stochastic Embedding

Depression is a prevalent mental illness that requires autonomous detection systems due to its complexity. Existing machine learning techniques face challenges such as background noise sensitivity, slow adaptation speed, and imbalanced data. To address these limitations, this study proposes a novel ModWave Cepstral Fusion and Stochastic Embedding Framework for depression prediction. Then, the Gain Modulated Wavelet Technique removes background noise and normalises audio signals. Difficulties with generalisation, which results in a lack of interpretability, hinder extracting relevant characteristics from speech. To address these issues, an Auto Cepstral Fusion extracts relevant features from speech, capturing temporal and spectral characteristics caused by background voice. Feature selection becomes imperative when choosing relevant features for classification. Selecting irrelevant features can result in overfitting, the curse of dimensionality, and less robustness to noise. Hence, the Principal Stochastic Embedding technique handles high-dimensional data, minimising noise influence and dimensionality. Furthermore, the XGBoost classifier differentiates between depressed and non-depressed individuals. As a result, the proposed method uses the DAIC-WOZ dataset from USC for detecting depressions, achieving an accuracy of 97.02%, precision of 97.02%, recall of 97.02%, F1-score of 97.02%, RMSE of 2.00, and MAE of 0.9, making it a promising tool for autonomous depression detection.

Stealthy Attack Against Distributed State Estimation for Cyber‐Physical Systems

ABSTRACTThis article studies a stealthy attack strategy design for distributed state estimation from the perspective of an attacker in cyber‐physical systems, where both internal and external attack scenarios are considered simultaneously. To enhance robustness against outliers and reduce communication burden, a distributed fusion estimator is developed by fusing the innovation residuals of neighboring smart sensor nodes without any attack. Based on the designed distributed fusion estimator, the evolution of the distributed estimation error covariance is analyzed, and its lower and upper bounds are obtained. Moreover, a stealthy attack framework embedding adjustable parameter is designed to weaken the estimation performance, where the constraints of the adjustable parameter are provided based on the desired attack effect. Finally, a simulation example is provided to manifest the validity of the main results.

Multi-view visual semantic embedding for cross-modal image–text retrieval

Visual Semantic Embedding (VSE) is a dominant method for cross-modal image–text retrieval. The purpose of VSE is to learn an embedding space where images can be embedded close to the corresponding captions. However, there are large intra-class variations in image–text data. Multiple captions describing the same image may be described from different views, and descriptions of different views are often dissimilar. The VSE method embeds samples from the same class in similar positions, which suppresses intra-class variations and leads to inferior generalization. This paper proposes a Multi-View Visual Semantic Embedding (MV-VSE) framework that learns multiple embeddings for an image, explicitly modeling intra-class variation. To optimize the MV-VSE framework, a multi-view triplet loss is proposed, which jointly optimizes multi-view embeddings while retaining intra-class variation. Recently, large-scale Vision-Language Pre-training (VLP) has become a new paradigm for cross-modal image–text retrieval. To allow our framework to be flexibly applied to the traditional VSE models and VSE-based VLP models, we incorporate the contrastive loss commonly used in VLP and the triplet loss into a unified loss, and further propose a multi-view unified loss. Our framework can be applied plug-and-play to traditional VSE models and VSE-based VLP models without excessively increasing model complexity. Experimental results on the image–text retrieval benchmark datasets demonstrate that applying our framework can boost the retrieval performance of current VSE models. The code is available at https://github.com/AAA-Zheng/MV-VSE.

GeoEntity-type constrained knowledge graph embedding for predicting natural-language spatial relations

Natural-language spatial relations between geographic entities (geoentities) reflect diverse perceptions influenced by factors like location, culture, and linguistic conventions. These relations play a crucial role in supporting geospatial tasks, such as question answering and cognitive reasoning. While prior studies focused on a limited set of human-selected spatial terms and geometric attributes, they often overlooked essential semantic attributes. To overcome this limitation, we developed a Spatial Relation-based Knowledge Graph Embedding framework, SR-KGE, with new KG fusion functions to predict spatial relation terms among distinct geoentities. This method not only considers graph structures and the diversity of natural language expressions in the embedding and learning process, but also incorporates geoentity types as a constraint to capture spatial and semantic relations more accurately. Our experiments on two knowledge graph datasets, one small-scale and one large-scale, have both shown its superior performance in spatial relation inference compared to popular KGE models, including TransE, RotatE, and HAKE. We hope our research will advance the classic study of natural language described spatial relations in a more automated and intelligent way.

Wasserstein task embedding for measuring task similarities

Measuring similarities between different tasks is critical in a broad spectrum of machine learning problems, including transfer, multi-task, continual, and meta-learning. Most current approaches to measuring task similarities are architecture-dependent: (1) relying on pre-trained models, or (2) training networks on tasks and using forward transfer as a proxy for task similarity. In this paper, we leverage the optimal transport theory and define a novel task embedding for supervised classification that is model-agnostic, training-free, and capable of handling (partially) disjoint label sets. In short, given a dataset with ground-truth labels, we perform a label embedding through multi-dimensional scaling and concatenate dataset samples with their corresponding label embeddings. Then, we define the distance between two datasets as the 2-Wasserstein distance between their updated samples. Lastly, we leverage the 2-Wasserstein embedding framework to embed tasks into a vector space in which the Euclidean distance between the embedded points approximates the proposed 2-Wasserstein distance between tasks. We show that the proposed embedding leads to a significantly faster comparison of tasks compared to related approaches like the Optimal Transport Dataset Distance (OTDD). Furthermore, we demonstrate the effectiveness of our embedding through various numerical experiments and show statistically significant correlations between our proposed distance and the forward and backward transfer among tasks on a wide variety of image recognition datasets.

Embedding constitutional rights

Abstract This article details the concept of constitutional embedding and demonstrates its utility in four country-rights cases. Constitutional embedding refers to the process by which some understanding of constitutional rights comes to take root in everyday life, moving from words on paper to something that shapes expectations and behavior. The degree of constitutional embedding varies along two dimensions: social and legal, or how individuals and groups operating in the social sphere understand and relate to constitutional rights, and how those working in the formal legal sphere do so. In a global political climate defined by democratic backsliding, powerful vested interests, and backlash against moves toward equality, the status of constitutional rights and how they become and remain embedded is doubly important. The constitutional embedding framework highlights how interactions between legal elites and ordinary citizens constitute the extent to which constitutional law influences daily life. The framework has broad applicability across contexts and rights domains.

Friend2User : A new CNN based method for user network and content embedding

Nowadays, social networks have become an integral part of modern society, significantly influencing individuals worldwide due to their extensive reach. Consequently, analyzing the data disseminated within these networks in order to identify online communities presents a major challenge for researchers in the data mining field. To address this challenge, we propose, in this paper, a novel deep user embedding framework for community extraction on social networks. Our method leverages the capability of Convolutional Neural Networks (CNNs) to produce abstract representations of users that preserve the semantic information in the data. Specifically, our approach considers both the profile content and the network structure, harnessing the power of unsupervised CNNs. The key concept underlying our proposal is that each user is represented not only by their own content but also by the content of their close friends. We employ a recursive CNN to integrate neighboring users’ content, thereby generating concise and informative user embeddings. The empirical findings obtained by our method demonstrate the effectiveness of our proposed user embeddings in efficiently detecting communities within social networks, particularly in the context of cybersecurity.

GFedKG: GNN-based federated embedding model for knowledge graph completion

Knowledge Graph (KG) completion through reasoning over KGs is one of the most concerned issue in research around knowledge graphs. Current work focuses more on studying individual KG, while a series of related multi-source KGs are more common in real applications. In order to effectively exert their complementary roles in completing each KG under the premise of ensuring data privacy, federated learning becomes the focus of attention. However, existing methods ignore the explicit complementarity among KGs in terms of structural information. And they generally align embeddings of overlapping instances across clients, which constrains flexibility of each client model. Based on this, we propose the first GNN-based federated embedding framework for multiple knowledge graph completion, GFedKG. It focuses on the complementary role among KGs in topology structures and adaptively generates different aggregated neighboring features for entities in distinct clients. Such treatment achieves some consistency among clients while retaining the flexibility of each client model to characterize respective KG. Moreover, entity embeddings that represent the features of entities are better privacy-preserved since neighboring embeddings are transmitted instead of them. And with the aim of realizing more refined and accurate aggregation of neighboring features from multiple clients, we innovatively propose a context-aware graph neural network, along with the context-enhanced evaluation function. Homophily neighbors are also taken into account to cope with the low-homophily aspect of KGs. Extensive experiments demonstrate the advantages of GFedKG.

PARE: A framework for removal of confounding effects from any distance-based dimension reduction method.

Dimension reduction tools preserving similarity and graph structure such as t-SNE and UMAP can capture complex biological patterns in high-dimensional data. However, these tools typically are not designed to separate effects of interest from unwanted effects due to confounders. We introduce the partial embedding (PARE) framework, which enables removal of confounders from any distance-based dimension reduction method. We then develop partial t-SNE and partial UMAP and apply these methods to genomic and neuroimaging data. For lower-dimensional visualization, our results show that the PARE framework can remove batch effects in single-cell sequencing data as well as separate clinical and technical variability in neuroimaging measures. We demonstrate that the PARE framework extends dimension reduction methods to highlight biological patterns of interest while effectively removing confounding effects.

Self-supervised multi-object tracking based on metric learning

The current paradigm of joint detection and tracking still requires a large amount of instance-level trajectory annotation, which incurs high annotation costs. Moreover, treating embedding training as a classification problem would lead to difficulties in model fitting. In this paper, we propose a new self-supervised multi-object tracking based on the real-time joint detection and embedding (JDE) framework, which we termed as self-supervised multi-object tracking (SS-MOT). In SS-MOT, the short-term temporal correlations between objects within and across adjacent video frames are both considered as self-supervised constraints, where the distances between different objects are enlarged while the distances between same object of adjacent frames are brought closer. In addition, short trajectories are formed by matching pairs of adjacent frames using a matching algorithm, and these matched pairs are treated as positive samples. The distances between positive samples are then minimized for futher the feature representation of the same object. Therefore, our method can be trained on videos without instance-level annotations. We apply our approach to state-of-the-art JDE models, such as FairMOT, Cstrack, and SiamMOT, and achieve comparable results to these supevised methods on the widely used MOT17 and MOT20 challenges.

Standard form of master equations for general non-Markovian jump processes: The Laplace-space embedding framework and asymptotic solution

We present a standard form of master equations (MEs) for general one-dimensional non-Markovian (history-dependent) jump processes, complemented by an asymptotic solution derived from an expanded system-size approach. The ME is obtained by developing a general Markovian embedding using a suitable set of auxiliary field variables. This Markovian embedding uses a Laplace-convolution operation applied to the velocity trajectory. We introduce an asymptotic method tailored for this ME standard, generalizing the system-size expansion for these jump processes. Under specific stability conditions tied to a single noise source, upon coarse graining, the generalized Langevin equation (GLE) emerges as a universal approximate model for point processes in the weak-coupling limit. This methodology offers a unified analytical tool set for general non-Markovian processes, reinforcing the universal applicability of the GLE founded in microdynamics and the principles of statistical physics. Published by the American Physical Society 2024

Autobalanced Multitask Node Embedding Framework for Intelligent Education.

Recently, online education has become popular. Many e-learning platforms have been launched with various intelligent services aimed at improving the learning efficiency and effectiveness of learners. Graphs are used to describe the pairwise relations between entities, and the node embedding technique is the foundation of many intelligent services, which have received increasing attention from researchers. However, the graph in the intelligent education scenario has three noteworthy properties, namely, heterogeneity, evolution, and lopsidedness, which makes it challenging to implement ecumenical node embedding methods on it. In this article, an autobalanced multitask node embedding model is proposed, named MNE, and applied to the interaction graph, settling a few actual tasks in intelligent education. More specifically, MNE builds two purpose-built self-supervised node embedding learning tasks for heterogeneous evolutive graphs. Edge-specific reconstruction tasks are built according to the semantic information and properties of the heterogeneous edges, and an evolutive weight regression task is designed, aiding the model to perceive the evolution of learners' implicit cognitive states. Then, both aleatoric and epistemic uncertainty quantification techniques are introduced, achieving both task- and node-level weight estimation and instructing subtask autobalancing. Experimental results on real-world datasets indicate that the proposed model outperforms the state-of-the-art graph embedding methods on two assessment tasks and demonstrates the validity of the proposed multitask framework and subtask balancing mechanism. Our implementations are available at https://github.com/ccnu-mathits/MNE4HEN.

AVBAE-MODFR: A novel deep learning framework of embedding and feature selection on multi-omics data for pan-cancer classification

Integration analysis of cancer multi-omics data for pan-cancer classification has the potential for clinical applications in various aspects such as tumor diagnosis, analyzing clinically significant features, and providing precision medicine. In these applications, the embedding and feature selection on high-dimensional multi-omics data is clinically necessary. Recently, deep learning algorithms become the most promising cancer multi-omic integration analysis methods, due to the powerful capability of capturing nonlinear relationships. Developing effective deep learning architectures for cancer multi-omics embedding and feature selection remains a challenge for researchers in view of high dimensionality and heterogeneity. In this paper, we propose a novel two-phase deep learning model named AVBAE-MODFR for pan-cancer classification. AVBAE-MODFR achieves embedding by a multi2multi autoencoder based on the adversarial variational Bayes method and further performs feature selection utilizing a dual-net-based feature ranking method. AVBAE-MODFR utilizes AVBAE to pre-train the network parameters, which improves the classification performance and enhances feature ranking stability in MODFR. Firstly, AVBAE learns high-quality representation among multiple omics features for unsupervised pan-cancer classification. We design an efficient discriminator architecture to distinguish the latent distributions for updating forward variational parameters. Secondly, we propose MODFR to simultaneously evaluate multi-omics feature importance for feature selection by training a designed multi2one selector network, where the efficient evaluation approach based on the average gradient of random mask subsets can avoid bias caused by input feature drift. We conduct experiments on the TCGA pan-cancer dataset and compare it with four state-of-the-art methods for each phase. The results show the superiority of AVBAE-MODFR over SOTA methods.

Collaborative Cross-Network Embedding Framework for Network Alignment

Collaborative Cross-Network Embedding Framework for Network Alignment

RiQ-KGC: Relation Instantiation Enhanced Quaternionic Attention for Complex-Relation Knowledge Graph Completion

A knowledge graph is a structured semantic network designed to describe physical entities and relations in the world. A comprehensive and accurate knowledge graph is essential for tasks such as knowledge inference and recommendation systems, making link prediction a popular problem for knowledge graph completion. However, existing approaches struggle to model complex relations among entities, which severely hampers their ability to complete knowledge graphs effectively. To address this challenge, we propose a novel hierarchical multi-head attention network embedding framework, called RiQ-KGC, which integrates different-grained contextual information of knowledge graph triples and models quaternion rotation relations between entities. Furthermore, we propose a relation instantiation method for alleviating the difficulty of expressing complex relations between entities. To enhance the expressiveness of relation representation, the relation is integrated by Transformer to obtain multi-hop neighbor information, so that one relation can be embedded into different embeddings according to different entities. Experimental results on four datasets demonstrate that RiQ-KGC exhibits strong competitiveness compared to state-of-the-art models in link prediction, while the ablation experiments reveal that the proposed relation instantiation method achieves great performance.

Sequence-based malware detection using a single-bidirectional graph embedding and multi-task learning framework

As an important part of malware detection and classification, sequence-based analysis can be integrated into dynamic detection system for real-time detection. This work presents a novel learning method for malware detection models that leverages advances in graph embedding for fusing the n-gram data into a one-hot feature space with different transmission directions. By capturing the information flow, our method finds a better feature representation for detection tasks with rely solely on sequence information. To enhance the stability of feature representation, this work adopts a multi-task learning strategy which achieves better performance in independent testing. We evaluate our method on two different realworld datasets and compare it against four superior malware detection models. During malware detection using our method, we conducted in-depth discussions on feature length, graph embedding direction, model depth, and different multi-task learning strategies. Experimental and discussion results show that our method significantly outperforms alternative approaches across evaluation settings.

A Survey on Reversible Data Hiding for Uncompressed Images

Reversible data hiding (RDH) has developed various theories and algorithms since the early 1990s. The existing works involve a large amount of specialized knowledge, making it difficult for researchers, especially primary learners, to have a good grounding in the basic ideas. In this survey, we will review the mainstream RDH algorithms in uncompressed images and analyze their unique features to provide readers with an introduction to basic topics in RDH. We analyze the most effective RDH frameworks and their common extensions. The classic techniques, including lossless compression-based RDH, difference expansion, integer transform, histogram shifting, prediction-error expansion (PEE), and their extensions, will be reviewed first. Then, three currently popular investigated schemes, i.e., multiple histograms modification, pairwise PEE, and pixel-value-ordering, are presented in detail. Four aspects of these mainstream techniques are reviewed and analyzed, including the evolution of embedding frameworks, detailed technological features, extensions, and the current state of the art. Furthermore, we look forward the possible future research based on early-age motivations.

TemporalHAN: Hierarchical attention-based heterogeneous temporal network embedding

Heterogeneous temporal network embedding aims to learn each node of different types of a heterogeneous temporal network in each snapshot into a low-dimensional vector representation, which can be used for various network analysis tasks such as node classification and relationship prediction. Our work proposes a novel heterogeneous temporal graph neural network embedding framework (TemporalHAN) based on hierarchical attention using a temporal convolutional network (TCN). In particular, we introduce node-level and semantic-level attention into heterogeneous graph neural networks to identify the importance of different levels between nodes. For each snapshot, we first utilise a new random walk algorithm (NRWA) to collect strongly connected heterogeneous neighbours for each node of different types and group them by node types. In addition, the algorithm utilises a damping factor to ensure that the more recent snapshots allocate more random walk steps. We then utilise node-level and semantic-level attention to learn the importance between a node and its random walk neighbour for a specific node type and learn the importance of different node-type for this node, respectively. Finally, we adopt TCN to capture the evolution information between snapshots. Experimental results on relationship prediction and node classification reveal that the TemporalHAN is competitive against diverse state-of-the-art approaches. Our code is available at https://github.com/Legendary-L/THAN.