Embedding Framework Research Articles

TSEE: a novel knowledge embedding framework for cyberspace security

TSEE: a novel knowledge embedding framework for cyberspace security

Emotion embedding framework with emotional self-attention mechanism for speaker recognition

The emotional states of speech have a great impact on the efficiency of speaker recognition (SR) system. Many researchers focus on how to map speech with different emotions to an emotion invariant embedding, which reduces the diversity of data. This paper proposes a new emotion embedding framework with self-attention mechanism for speaker recognition. First, several deep neural networks (DNNs) are trained to classify speakers in different emotional states as emotion embedding extractors during development phase. Then at enrollment stage, these pre-trained models are used to extend emotion embeddings from neutral speech. In order to make the final speaker embedding more representative, the classification model is trained with self-attention mechanism in emotion dimension, so that the framework can automatically annotate the weights of the emotion embeddings. Experiments were carried out on both Mandarin Affective Speech Corpus (MASC) and Crowd-Sourced Emotional Multimodal Actors Dataset (CREMA-D). The results show the proposed method achieves the best of Identification Rate (IR) and Equal Error Rate (EER) which are 59.14%, 15.79% on MASC and 75.98%, 8.14% on CREMA-D compared with state-of-the-art methods. In addition, the cross-database experiments also further demonstrate the practicability of the method in real scenes.

A generic framework for embedding human brain function with temporally correlated autoencoder.

Learning an effective and compact representation of human brain function from high-dimensional fMRI data is crucial for studying the brain's functional organization. Traditional representation methods such as independent component analysis (ICA) and sparse dictionary learning (SDL) mainly rely on matrix decomposition which represents the brain function as spatial brain networks and the corresponding temporal patterns. The correspondence of those brain networks across individuals are built by viewing them as one-hot vectors and then performing the matching. However, those one-hot vectors do not encode the regularity and/or variability of different brains very well, and thus are limited in effectively representing the functional brain activities across individuals and among different time points. To address this problem, in this paper, we formulate the human brain functional representation as an embedding problem, and propose a novel embedding framework based on the Transformer model to encode the brain function in a compact, stereotyped and comparable latent space where the brain activities are represented as dense embedding vectors. We evaluate the proposed embedding framework on the publicly available Human Connectome Project (HCP) task fMRI dataset. The experiments on brain state prediction task indicate the effectiveness and generalizability of the learned embedding. We also explore the interpretability of the learned embedding from both spatial and temporal perspective. In general, our approach provides novel insights on representing the regularity and variability of human brain function in a general, comparable, and stereotyped latent space.

Look-Ahead VNF-FG Embedding Framework for Latency-Sensitive Network Services

Dynamic and zero-touch management is expected to be the key feature of next-generation 6G networks. Network Function Virtualization (NFV) is one of the key technologies for realizing such management through software-based networks. Despite great benefits offered by NFV, deploying network services (NSs) in NFV ecosystems remains a challenge, especially for latency-sensitive NSs, as they demand stringent latency requirements and fast service provisioning. Specifically, service graphs should be embedded into an infrastructure such that these requirements are satisfied while optimizing network operator’s objectives. To cope with the scalability of optimization-based approaches, heuristic methods are known as promising alternatives to find a satisfactory solution within an acceptable execution time. However, existing VNF embedding heuristics still suffer from the so-called causality issue, which may degrade the embedding solution quality. The causality issue means that embedding decisions cannot be optimally determined before all neighboring dependencies are known. To this end, we introduce our h-horizon sequential look-ahead greedy embedding framework, which provides efficient embedding and re-embedding strategies to alleviate the impact of the causality issue. The simulation results indicate that our proposed algorithm significantly improves embedding cost, compared to the existing heuristic algorithms while being much more scalable than an optimization-based approach.

Zone-Enhanced Spatio-Temporal Representation Learning for Urban POI Recommendation

Points-of-interest (POIs) recommendation plays a vital role in location-based social networks (LBSNs) by introducing unexplored POIs to consumers and has drawn extensive attention from academia and industry. Existing POI recommender systems usually learn fixed latent vectors to represent both consumers and POIs from historical check-ins and make recommendations under the spatio-temporal constraints. However, we argue that the existing works still suffer from the challenges of explaining consumers' complicated check-in actions. To this end, we first explore the interpretability of recommendations from the POI aspect, i.e., for a specific POI, its function usually changes over time, so representing a POI with a single fixed latent vector is not sufficient to describe the dynamic nature of POIs. Besides, check-in actions to a POI are also affected by the zone where it is located. In other words, the zone's embedding learned from POI distributions, road segments, and historical check-ins could be jointly utilized to enhance POI embeddings. Along this line, we propose a <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">T</b> ime-z <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">o</b> ne-space <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">P</b> OI embedding model ( <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ToP</b> ), which integrates multi-knowledge graphs and topic model to introduce not only spatio-temporal effects but also sentiment constraints into POI embeddings for strengthening interpretability of recommendation. Specifically, ToP learns multiple latent vectors for a POI in a different period with spatial constraints via knowledge graph learning. To add sentiment constraints, ToP jointly combines these vectors with the zone's representations learned by topic models to make explainable recommendations. ToP considers the time, space, and sentiment of POI in a unified embedding framework, which benefits the POI recommendations. Extensive experiments on real-world Changchun city datasets demonstrate that ToP achieves state-of-the-art performance in terms of common metrics and provides more insights for consumers' POI check-in actions.

SSKGE: a time-saving knowledge graph embedding framework based on structure enhancement and semantic guidance

SSKGE: a time-saving knowledge graph embedding framework based on structure enhancement and semantic guidance

A type-augmented knowledge graph embedding framework for knowledge graph completion

Knowledge graphs (KGs) are of great importance to many artificial intelligence applications, but they usually suffer from the incomplete problem. Knowledge graph embedding (KGE), which aims to represent entities and relations in low-dimensional continuous vector spaces, has been proved to be a promising approach for KG completion. Traditional KGE methods only concentrate on structured triples, while paying less attention to the type information of entities. In fact, incorporating entity types into embedding learning could further improve the performance of KG completion. To this end, we propose a universal Type-augmented Knowledge graph Embedding framework (TaKE) which could utilize type features to enhance any traditional KGE models. TaKE automatically captures type features under no explicit type information supervision. And by learning different type representations of each entity, TaKE could distinguish the diversity of types specific to distinct relations. We also design a new type-constrained negative sampling strategy to construct more effective negative samples for the training process. Extensive experiments on four datasets from three real-world KGs (Freebase, WordNet and YAGO) demonstrate the merits of our proposed framework. In particular, combining TaKE with the recent tensor factorization KGE model SimplE can achieve state-of-the-art performance on the KG completion task.

Network embedding framework for driver gene discovery by combining functional and structural information

Comprehensive analysis of multiple data sets can identify potential driver genes for various cancers. In recent years, driver gene discovery based on massive mutation data and gene interaction networks has attracted increasing attention, but there is still a need to explore combining functional and structural information of genes in protein interaction networks to identify driver genes. Therefore, we propose a network embedding framework combining functional and structural information to identify driver genes. Firstly, we combine the mutation data and gene interaction networks to construct mutation integration network using network propagation algorithm. Secondly, the struc2vec model is used for extracting gene features from the mutation integration network, which contains both gene's functional and structural information. Finally, machine learning algorithms are utilized to identify the driver genes. Compared with the previous four excellent methods, our method can find gene pairs that are distant from each other through structural similarities and has better performance in identifying driver genes for 12 cancers in the cancer genome atlas. At the same time, we also conduct a comparative analysis of three gene interaction networks, three gene standard sets, and five machine learning algorithms. Our framework provides a new perspective for feature selection to identify novel driver genes.

A flexible PageRank-based graph embedding framework closely related to spectral eigenvector embeddings

A flexible PageRank-based graph embedding framework closely related to spectral eigenvector embeddings

SEPAKE: a structure-enhanced and position-aware knowledge embedding framework for knowledge graph completion

SEPAKE: a structure-enhanced and position-aware knowledge embedding framework for knowledge graph completion

A Provable Framework of Learning Graph Embeddings via Summarization

Given a large graph, can we learn its node embeddings from a smaller summary graph? What is the relationship between embeddings learned from original graphs and their summary graphs? Graph representation learning plays an important role in many graph mining applications, but learning em-beddings of large-scale graphs remains a challenge. Recent works try to alleviate it via graph summarization, which typ-ically includes the three steps: reducing the graph size by combining nodes and edges into supernodes and superedges,learning the supernode embedding on the summary graph and then restoring the embeddings of the original nodes. How-ever, the justification behind those steps is still unknown. In this work, we propose GELSUMM, a well-formulated graph embedding learning framework based on graph sum-marization, in which we show the theoretical ground of learn-ing from summary graphs and the restoration with the three well-known graph embedding approaches in a closed form.Through extensive experiments on real-world datasets, we demonstrate that our methods can learn graph embeddings with matching or better performance on downstream tasks.This work provides theoretical analysis for learning node em-beddings via summarization and helps explain and under-stand the mechanism of the existing works.

Task-Specific Scene Structure Representations

Understanding the informative structures of scenes is essential for low-level vision tasks. Unfortunately, it is difficult to obtain a concrete visual definition of the informative structures because influences of visual features are task-specific. In this paper, we propose a single general neural network architecture for extracting task-specific structure guidance for scenes. To do this, we first analyze traditional spectral clustering methods, which computes a set of eigenvectors to model a segmented graph forming small compact structures on image domains. We then unfold the traditional graph-partitioning problem into a learnable network, named Scene Structure Guidance Network (SSGNet), to represent the task-specific informative structures. The SSGNet yields a set of coefficients of eigenvectors that produces explicit feature representations of image structures. In addition, our SSGNet is light-weight (56K parameters), and can be used as a plug-and-play module for off-the-shelf architectures. We optimize the SSGNet without any supervision by proposing two novel training losses that enforce task-specific scene structure generation during training. Our main contribution is to show that such a simple network can achieve state-of-the-art results for several low-level vision applications including joint upsampling and image denoising. We also demonstrate that our SSGNet generalizes well on unseen datasets, compared to existing methods which use structural embedding frameworks. Our source codes are available at https://github.com/jsshin98/SSGNet.

A new method to assess sustainability of agricultural systems by integrating ecosystem services and environmental impacts

Agricultural stakeholders disagree on how to put the general concept of sustainability into practice. But most recognize the need for assessment methods to characterize the current situation and gauge progress yet to be made. Many sustainability assessment methods for agricultural systems have been published, but very few integrate ecosystem services (ES). And yet, ES indicators are key to better integrating biological processes to support the agroecological transition. Although researchers have attempted to link the concept of environmental impact (EI) and ES through life cycle analysis (LCA), gaps remain when it comes to doing so within a sustainability assessment method. Here a new method based on an original conceptual indicator framework embedding new predictive indicators of EI and ES indicators was developed. This framework covers 22 environmental themes and integrates few basic economic and social indicators. The method was implemented on 33 farms with different certifications and production specifications. All farms were members of an agricultural cooperative in north-eastern France. We selected 250 plots to cover the farms’ crop rotations. The correlation analysis showed no systemic trade-off between EI or ES and economic indicators. Meanwhile, the joint assessment of ES and EI (like the multivariate analysis), delivered more nuanced results than a classical opposition between intensive and organic farming. Some farms with a moderate or even good level of ES often do not leverage them, as shown by their high EI. The principle component analysis and subsequent hierarchical ascendant classification allowed us to distinguish five clusters providing a more complex picture than simply opposing conventional and organic agriculture. Finally, the positive deviance analysis revealed that economic, environmental and social considerations can be balanced in a way that seems to be more sustainable than, for example, an organic farm that has very successfully reduced EI but at a cost of several economic and social trade-offs.

Semantic-guided graph neural network for heterogeneous graph embedding

Heterogeneous Graph Neural Network (HGNN) has shown a great promising in embedding complex structural and semantic information of heterogeneous graph. However, many popular HGNNs fail to capture the meaningful characteristics to distinguish heterogeneous nodes, known as the semantic confusion problem. In this paper, we hold that semantic confusion problem can be addressed by jumping knowledge toward semantics enrichment, and propose a general framework for heterogeneous graph embedding named Semantic-guided Graph Neural Network (SGNN). Here, we develop novel two-level fusion mechanisms in both node and semantic aggregation. In the node-level, we aggregate the local neighbors with jumping knowledge to learn an enhanced local representation. In the semantic-level, we maximize the common representation to extract the jumping knowledge from multiple semantics in latent space. The common representation is injected into semantic-level aggregation as jumping knowledge to guide the model to pay more attention on target semantics. In the end, the semantic confusion problem is shown to be alleviated in the two-level semantic-guided aggregation framework of SGNN through both theory and experiments. Experimental results demonstrate that SGNN gains highest results in real-world tasks. We also perform validation experiments to evaluate the effectiveness of aggregation mechanism towards semantic confusion in each level, and the results show superiority of SGNN.

PME: pruning-based multi-size embedding for recommender systems.

Embedding is widely used in recommendation models to learn feature representations. However, the traditional embedding technique that assigns a fixed size to all categorical features may be suboptimal due to the following reasons. In recommendation domain, the majority of categorical features' embeddings can be trained with less capacity without impacting model performance, thereby storing embeddings with equal length may incur unnecessary memory usage. Existing work that tries to allocate customized sizes for each feature usually either simply scales the embedding size with feature's popularity or formulates this size allocation problem as an architecture selection problem. Unfortunately, most of these methods either have large performance drop or incur significant extra time cost for searching proper embedding sizes. In this article, instead of formulating the size allocation problem as an architecture selection problem, we approach the problem from a pruning perspective and propose Pruning-based Multi-size Embedding (PME) framework. During the search phase, we prune the dimensions that have the least impact on model performance in the embedding to reduce its capacity. Then, we show that the customized size of each token can be obtained by transferring the capacity of its pruned embedding with significant less search cost. Experimental results validate that PME can efficiently find proper sizes and hence achieve strong performance while significantly reducing the number of parameters in the embedding layer.

A large-scale data security detection method based on continuous time graph embedding framework

Graph representation learning has made significant strides in various fields, including sociology and biology, in recent years. However, the majority of research has focused on static graphs, neglecting the temporality and continuity of edges in dynamic graphs. Furthermore, dynamic data are vulnerable to various security threats, such as data privacy breaches and confidentiality attacks. To tackle this issue, the present paper proposes a data security detection method based on a continuous-time graph embedding framework (CTDGE). The framework models temporal dependencies and embeds data using a graph representation learning method. A machine learning algorithm is then employed to classify and predict the embedded data to detect if it is secure or not. Experimental results show that this method performs well in data security detection, surpassing several dynamic graph embedding methods by 5% in terms of AUC metrics. Furthermore, the proposed framework outperforms other dynamic baseline methods in the node classification task of large-scale graphs containing 4321477 temporal information edges, resulting in a 10% improvement in the F1 score metric. The framework is also robust and scalable for application in various data security domains. This work is important for promoting the use of continuous-time graph embedding framework in the field of data security.

A novel employability embedding framework for three-year bachelor’s programs

Employability skills and capabilities are equally important as academic and technical knowledge in the current job market. Consequently, universities are increasingly focusing on providing employability skills to their students. While universities in Australia are experimenting with various methods for employability embedding, we could not find any framework for embedding employability at the course level. Mapping employability at the course level is non-trivial but necessary to scaffold employability. e-portfolios form an integral part of employability; however, it is not well defined in the literature how to embed and assess an e-portfolio into a tertiary academic program. Unless an e-portfolio is embedded and assessed in the academic program, the uptake by the students cannot be guaranteed; hence students miss out on the benefits of such an important tool. This paper addresses these two key challenges. It develops an employability embedding framework targeted for a three-year bachelor’s program. This framework also enables the embedding and assessment of e-portfolios. The framework is novel as it offers a scaffolded and organic way to embed employability

Employing Latent Categories of Entities for Knowledge Graph Embeddings With Contrastive Learning

Knowledge graph embedding aims to represent entities and relations as low-dimensional vectors, which is an effective way for robotics to learn and reason about semantic knowledge. It is crucial for knowledge graph embedding models to infer various relation patterns, such as symmetry/antisymmetry. However, most existing approaches ignore the latent semantic categories information of entities. For example, when robots use a faucet with relation <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">OperatesOn</i> , its tail entity category should be washable objects, such as apple, bowl, rather than non-washable objects, such as mobile phone, computer. To address such issues, we propose a novel and simple framework for knowledge graph embedding, which utilizes contrastive learning to cluster entities based on the constraints arising from relations, and enhances the discriminative ability for entities with the latent same category. Experimental results demonstrate the effectiveness of our proposed method on four standard knowledge graph completion benchmarks. It is noteworthy that our method can yield some new state-of-the-art results, achieving 88.2% MRR, 84.9% Hits@1 on the AI2Thor dataset, and 39.8% MRR, 30.0% Hits@1 on the FB15k-237 dataset.

Sem2vec : Semantics-aware Assembly Tracelet Embedding

Binary code similarity is the foundation of many security and software engineering applications. Recent works leverage deep neural networks (DNN) to learn a numeric vector representation (namely, embeddings ) of assembly functions, enabling similarity analysis in the numeric space. However, existing DNN-based techniques capture syntactic-, control flow-, or data flow-level information of assembly code, which is too coarse-grained to represent program functionality. These methods can suffer from low robustness to challenging settings such as compiler optimizations and obfuscations. We present sem2vec , a binary code embedding framework that learns from semantics . Given the control-flow graph (CFG), 34 pages. of an assembly function, we divide it into tracelets , denoting continuous and short execution traces that are reachable from the function entry point. We use symbolic execution to extract symbolic constraints and other auxiliary information on each tracelet. We then train masked language models to compute embeddings of symbolic execution outputs. Last, we use graph neural networks, to aggregate tracelet embeddings into the CFG-level embedding for a function. Our evaluation shows that sem2vec extracts high-quality embedding and is robust against different compilers, optimizations, architectures, and popular obfuscation methods including virtualization obfuscation. We further augment a vulnerability search application with embeddings computed by sem2vec and demonstrate a significant improvement in vulnerability search accuracy.

Unified robust network embedding framework for community detection via extreme adversarial attacks

Graph data are widely available in complex network systems. Numerous community detection algorithms have been investigated to study graph problems wherein the network topology offers abundant behavioral and functional information. In real scenarios, networks are extremely susceptible to external perturbations, primarily because of sparse topological information. Consequently, maintaining robust community detection performance when confronted with intricate network attacks or perturbations is challenging. Previous anti-graph perturbation methods have typically been adjusted to specific perturbation types, leading to the degradation or failure of these defences when confronted with other types of attack. Accordingly, a unified robust framework that leverages extreme adversarial attacks is proposed in this paper. Specifically, a novel graph perturbation module is introduced into the model to generate extreme undirected or directed perturbations through dynamic updating; this ensures that the model can fit the attacked network well. The proposed framework derives a unified perturbation formulation capable of simultaneously attacking symmetric and asymmetric networks. Furthermore, this framework can be applied to many existing non-negative matrix factorization-based community detection methods. Extensive experiments on artificial and real-world networks demonstrate that the proposed framework significantly improves the robustness of detection tasks, particularly in noisy networks.