Learning Network Representations Research Articles

DyHNet: Learning dynamic heterogeneous network representations

Many real-world networks, such as social networks, contain structural heterogeneity and experience temporal evolution. However, while there has been growing literature on network representation learning, only a few have addressed the need to learn representations for dynamic heterogeneous networks. The objective of our work in this paper is to introduce DyHNet, which learns representations for such networks and distinguishes itself from the state-of-the-art by systematically capturing (1) local node semantics, (2) global network semantics and (3) longer-range temporal associations between network snapshots when learning network representations. Through experiments on four real-world datasets from different domains, namely IMDB with 4,178 movies, AMiner with 10,674 papers, Yelp with 2,693 businesses, and DBLP with 14,376 papers, we demonstrate that our proposed method is able to show consistently better and more robust performance compared to the state-of-the-art techniques on link prediction and node classification tasks. More specifically, we are superior to the best baseline in the temporal link prediction task by approximately 13% and 15% on F1-score for the IMDB and AMiner datasets, respectively. Further, in the node classification task, our findings illustrate that the Micro F1 scores of our proposed model increase by 13% and 17% compared to the runner-up model on the Yelp and DBLP datasets, respectively.1

Amer: A New Attribute-Missing Network Embedding Approach.

Network embedding which aims to learn a low dimensional representation of nodes is a powerful technique for network analysis. While network embedding for networks with complete attributes has been widely investigated, in many real-world applications the attributes of partial nodes are unobserved (i.e., missing) due to privacy concern or resource limit. Very recently, several network embedding methods have been proposed for attribute-missing networks. They first complete the missing attributes and then use the complemented network to learn network embedding. The parameters of these two processes cannot be adjusted by each other, resulting in compromised results. To address this problem, we propose a unified model in which the process of completing missing attributes and the process of learning embedding are not separated but closely intertwined. Being specific, completing missing attributes is under the guidance of learning network representation via mutual information maximization, and the complemented attributes directly enter network representation module which will generate further feedback for completing missing attributes. We further impose attribute-structure relationship constraint for completing missing attributes by designing a new generative adversarial networks (GANs) model. To the best of our knowledge, this is the first unified model for attribute-missing network embedding. Empirical results on real-world datasets show the superiority of our new method over other state-of-the-art methods on four network analysis tasks, including node classification, node clustering, link prediction, and network visualization.

Privacy-Preserving Representation Learning for Text-Attributed Networks with Simplicial Complexes

Although recent network representation learning (NRL) works in text-attributed networks demonstrated superior performance for various graph inference tasks, learning network representations could always raise privacy concerns when nodes represent people or human-related variables. Moreover, standard NRLs that leverage structural information from a graph proceed by first encoding pairwise relationships into learned representations and then analysing its properties. This approach is fundamentally misaligned with problems where the relationships involve multiple points, and topological structure must be encoded beyond pairwise interactions. Fortunately, the machinery of topological data analysis (TDA) and, in particular, simplicial neural networks (SNNs) offer a mathematically rigorous framework to evaluate not only higher-order interactions, but also global invariant features of the observed graph to systematically learn topological structures. It is critical to investigate if the representation outputs from SNNs are more vulnerable compared to regular representation outputs from graph neural networks (GNNs) via pairwise interactions. In my dissertation, I will first study learning the representations with text attributes for simplicial complexes (RT4SC) via SNNs. Then, I will conduct research on two potential attacks on the representation outputs from SNNs: (1) membership inference attack, which infers whether a certain node of a graph is inside the training data of the GNN model; and (2) graph reconstruction attacks, which infer the confidential edges of a text-attributed network. Finally, I will study a privacy-preserving deterministic differentially private alternating direction method of multiplier to learn secure representation outputs from SNNs that capture multi-scale relationships and facilitate the passage from local structure to global invariant features on text-attributed networks.

MARML: Motif-Aware Deep Representation Learning in Multilayer Networks.

The rapid increase in high-throughput, complex, and heterogeneous data has led to the adoption of network-structured models and analyses for interpretation. However, these data are inherently complex and challenging to understand, prompting researchers to turn to graph embedding methods to facilitate analysis. While general network embedding techniques have shown promise in improving downstream prediction and classification tasks, real-world data are complicated due to cross-domain interactions between different types of entities. Multilayered networks have been successful in integrating biological data to represent biological systems' hierarchy, but embedding nodes based on different types of interactions remains an unsolved problem. To address this challenge, we propose the Motif-aware deep representation learning in multilayer (MARML) networks for learning network representations. Our method considers recurring motif patterns, topological information, and attributive information from other sources as node features. We validated the MARML method using various multilayer network datasets. In addition, by incorporating motif information, MARML considers higher order connections across different hierarchies. The learned features exhibited excellent accuracy in tasks related to link prediction and link differentiation, enabling us to distinguish between existing and disconnected triplets. Through the integration of both intrinsic node attributes and topological network structures, we enhance our understanding of complex biological systems.

Network representation learning based on social similarities

Analysis of large-scale networks generally requires mapping high-dimensional network data to a low-dimensional space. We thus need to represent the node and connections accurate and effectively, and representation learning could be a promising method. In this paper, we investigate a novel social similarity-based method for learning network representations. We first introduce neighborhood structural features for representing node identities based on higher-order structural parameters. Then the node representations are learned by a random walk approach that based on the structural features. Our proposed truss2vec is able to maintain both structural similarity of nodes and domain similarity. Extensive experiments have shown that our model outperforms the state-of-the-art solutions.

Lime: Low-Cost and Incremental Learning for Dynamic Heterogeneous Information Networks

Understanding the interconnected relationships of large-scale information networks like social, scholar and Internet of Things networks is vital for tasks like recommendation and fraud detection. The vast majority of the real-world networks are inherently heterogeneous and dynamic, containing many different types of nodes and edges and can change drastically over time. The dynamicity and heterogeneity make it extremely challenging to reason about the network structure. Unfortunately, existing approaches are inadequate in modeling real-life dynamical networks as they either have strong assumption of a given stochastic process or fail to capture the heterogeneity of network structure, and they all require extensive computational resources. We introduce <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Lime</small> , a better approach for modeling dynamic and heterogeneous information networks. <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Lime</small> is designed to extract high-quality network representation with significantly lower memory resources and computational time over the state-of-the-arts. Unlike prior work that uses a vector to encode each network node, we exploit the semantic relationships among network nodes to encode multiple nodes with similar semantics in shared vectors. By using many fewer node vectors, our approach significantly reduces the required memory space for encoding large-scale networks. To effectively trade information sharing for reduced memory footprint, we employ the recursive neural network (RsNN) with carefully designed optimization strategies to explore the node semantics in a novel cuboid space. We then go further by showing, for the first time, how an effective incremental learning approach can be developed – with the help of RsNN, our cuboid structure, and a set of novel optimization techniques – to allow a learning framework to quickly and efficiently adapt to a constantly evolving network. We evaluate <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Lime</small> by applying it to three representative network-based tasks, node classification, node clustering and anomaly detection, performing on three large-scale datasets. We compare <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Lime</small> against eleven prior state-of-the-art approaches for learning network representation. Our extensive experiments demonstrate that <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Lime</small> not only reduces the memory footprint by over 80 percent and the processing time over 2x when learning network representation but also delivers comparable performance for downstream processing tasks. We show that our incremental learning method can boost the learning time by up to 20x without compromising the quality of the learned network representation.

A Mutual Multi-Scale Triplet Graph Convolutional Network for Classification of Brain Disorders Using Functional or Structural Connectivity.

Brain connectivity alterations associated with mental disorders have been widely reported in both functional MRI (fMRI) and diffusion MRI (dMRI). However, extracting useful information from the vast amount of information afforded by brain networks remains a great challenge. Capturing network topology, graph convolutional networks (GCNs) have demonstrated to be superior in learning network representations tailored for identifying specific brain disorders. Existing graph construction techniques generally rely on a specific brain parcellation to define regions-of-interest (ROIs) to construct networks, often limiting the analysis into a single spatial scale. In addition, most methods focus on the pairwise relationships between the ROIs and ignore high-order associations between subjects. In this letter, we propose a mutual multi-scale triplet graph convolutional network (MMTGCN) to analyze functional and structural connectivity for brain disorder diagnosis. We first employ several templates with different scales of ROI parcellation to construct coarse-to-fine brain connectivity networks for each subject. Then, a triplet GCN (TGCN) module is developed to learn functional/structural representations of brain connectivity networks at each scale, with the triplet relationship among subjects explicitly incorporated into the learning process. Finally, we propose a template mutual learning strategy to train different scale TGCNs collaboratively for disease classification. Experimental results on 1,160 subjects from three datasets with fMRI or dMRI data demonstrate that our MMTGCN outperforms several state-of-the-art methods in identifying three types of brain disorders.

Bridging multimedia heterogeneity gap via Graph Representation Learning for cross-modal retrieval

Information retrieval among different modalities becomes a significant issue with many promising applications. However, inconsistent feature representation of various multimedia data causes the “heterogeneity gap” among various modalities, which is a challenge in cross-modal retrieval. For bridging the “heterogeneity gap,” the popular methods attempt to project the original data into a common representation space, which needs great fitting ability of the model. To address the above issue, we propose a novel Graph Representation Learning (GRL) method for bridging the heterogeneity gap, which does not project the original feature into an aligned representation space but adopts a cross-modal graph to link different modalities. The GRL approach consists of two subnetworks, Feature Transfer Learning Network (FTLN) and Graph Representation Learning Network (GRLN). Firstly, FTLN model finds a latent space for each modality, where the cosine similarity is suitable to describe their similarity. Then, we build a cross-modal graph to reconstruct the original data and their relationships. Finally, we abandon the features in the latent space and turn into embedding the graph vertexes into a common representation space directly. During the process, the proposed Graph Representation Learning method bypasses the most challenging issue by utilizing a cross-modal graph as a bridge to link the “heterogeneity gap” among different modalities. This attempt utilizes a cross-modal graph as an intermediary agent to bridge the “heterogeneity gap” in cross-modal retrieval, which is simple but effective. Extensive experiment results on six widely-used datasets indicate that the proposed GRL outperforms other state-of-the-art cross-modal retrieval methods.

Coupling network vertex representation learning based on network embedding method

Network representation learning is a basic problem in network data analysis. By learning network representation vectors, network vertices can be represented more accurately. With the development of deep learning, embedding methods have been widely used for network vertex representation learning. Providing that network data have changed in terms of their scale and modality, the research focus gradually shifted from single network mining to coupling network mining. This paper first analyzes the research status of embedding methods for single networks and then compares their advantages and disadvantages. Furthermore, the paper presents a model called CWCNE for coupling network embedding. The random walk and training algorithms of the proposed model are improved to adapt to coupling network features. The validity of the proposed model was verified using social, academic, film, poetry, and work coupling network data. Good results were obtained on community detection, entity recognition, and label classification tasks.

Learning Network Representations With Different Order Structural Information

Network embeddings aim to learn representations of nodes in a network with both the first- and the high-order proximities preserved. The first-order proximity corresponds to network reconstruction, while the high-order proximity is in tune with network inference. Since the tradeoff between the two proximities varies on scenarios, we propose an adjustable network embedding (ANE) algorithm for adjusting the weight between the first- and the high-order proximities. ANE is based on two hypotheses: 1) nodes in closed triplets are more important than nodes in open triplets and 2) closed triplets with higher degrees are more important. In addition, we change the bidirectional sampling of Word2vec into directional sampling to preserve the frequency of node pairs in the training set. Three common tasks, network reconstruction, link prediction, and classification are conducted on various publicly available data sets to validate the abovementioned statements.

PSO-ANE: Adaptive Network Embedding With Particle Swarm Optimization

Network embedding plays an important role in various network applications, such as node classification and link prediction. Lots of structure-based network embedding methods have been proposed. Yet, they suffer from unsteady embedding performances due to the parameter sensitivity. How to extract valuable attribute information in networks with less parameter influence is still a challenge. In this paper, we propose a novel algorithm, named adaptive network embedding with particle swarm optimization (PSO-ANE), which is based on the second-order dynamic random walk and PSO method, for learning network representations. A second-order dynamic random walk is designed to search a suitable strategy for each node based on the structure-based transition probability, the centrality-based transition probability, and the static link weights. To reduce the parameter dependence, PSO is adopted for key-parameter optimization to get global steady network embedding. The experiments validate that the proposed method outperforms the existing state-of-the-art techniques on multilabel classification, multiclass classification, and link prediction tasks.

AspEm: Embedding Learning by Aspects in Heterogeneous Information Networks.

Heterogeneous information networks (HINs) are ubiquitous in real-world applications. Due to the heterogeneity in HINs, the typed edges may not fully align with each other. In order to capture the semantic subtlety, we propose the concept of aspects with each aspect being a unit representing one underlying semantic facet. Meanwhile, network embedding has emerged as a powerful method for learning network representation, where the learned embedding can be used as features in various downstream applications. Therefore, we are motivated to propose a novel embedding learning framework-ASPEM-to preserve the semantic information in HINs based on multiple aspects. Instead of preserving information of the network in one semantic space, ASPEM encapsulates information regarding each aspect individually. In order to select aspects for embedding purpose, we further devise a solution for ASPEM based on dataset-wide statistics. To corroborate the efficacy of ASPEM, we conducted experiments on two real-words datasets with two types of applications-classification and link prediction. Experiment results demonstrate that ASPEM can outperform baseline network embedding learning methods by considering multiple aspects, where the aspects can be selected from the given HIN in an unsupervised manner.

Learning network representations

In this review I present several representation learning methods, and discuss the latest advancements with emphasis in applications to network science. Representation learning is a set of techniques that has the goal of efficiently mapping data structures into convenient latent spaces. Either for dimensionality reduction or for gaining semantic content, this type of feature embeddings has demonstrated to be useful, for example, for node classification or link prediction tasks, among many other relevant applications to networks. I provide a description of the state-of-the-art of network representation learning as well as a detailed account of the connections with other fields of study such as continuous word embeddings and deep learning architectures. Finally, I provide a broad view of several applications of these techniques to networks in various domains.