Representation Learning Model Research Articles

Unveiling hierarchical relationships for social image representation learning

Amidst the rapid propagation of images across social media platforms, the representation learning of social images has emerged as a prime research pursuit. A salient characteristic of social images lies in their multimodal properties, encompassing the visual content and textual descriptions present within individual images and the social relationships among different images. Notably, the content information within individual images and the structural information among different images exhibit a hierarchical nature. However, existing endeavors generally employ a flat framework, leading to suboptimal utilization of hierarchical relationships. Furthermore, the heterogeneity inherent in data content and structural information presents additional challenges in social image representation. In light of these challenges, we propose a novel Hierarchical Heterogeneous Graph Neural Network model for Social Image Representation learning, dubbed HHGSI. Our motivation lies in exploring and exploiting the hierarchical relationship between diverse modalities through designing the hierarchical heterogeneous network framework. HHGSI consists of an Intra-node Multimodal Graph Encoder and an Inter-node Heterogeneous Graph Neural Network to simultaneously capture fine-grained correlation within the image and heterogeneous relationships among the images. Moreover, a task-independent optimization objective is designed to make the model suitable for numerous network-oriented and multimodal tasks. Our proposal is extensively evaluated over four real-world datasets, and experimental results demonstrate the superiority of our proposal. Our code is publicly available in https://github.com/multimodal-code/HHGSI.

Multi-relational dynamic graph representation learning

In recent years, many dynamic graph representation learning methods have emerged due to the ubiquity of dynamic graph networks, such as social networks, medical networks, citation networks and traffic networks. Many researchers only consider the unitary relational topological information in a dynamic graph. However, real dynamic networks contain a large amount of multi-relational topological information. For example, there are different interactive relations such as sending a message, adding a friend, making a phone call, and sending an email in a social network, and they have different effects on node representation and should be distinguished. In addition, the non-topological information of nodes plays an important role in the node representation. Although these two types of information have been shown to improve the performance of many dynamic graph tasks, existing dynamic graph representation learning models could not integrate them well. Therefore, in this paper, we propose MRDGNN, a Multi-Relational Dynamic Graph Neural Network model, which can capture the dynamic evolution under each relational topology in the graph through a temporal multi-relational topology updater, including the participation of multi-relational topological and non-topological information of nodes. These two kinds of information will be adaptively fused into the representation of nodes by a merger. MRDGNN is continuously updated with the evolution of dynamic graphs and is a real-time learnable representation learning framework. Finally, we validate the effectiveness of MRDGNN for link prediction and relation prediction on four real datasets.

HAT-GAE: Self-supervised graph autoencoders with hierarchical adaptive masking and trainable corruption

In recent years, self-supervised autoencoders have developed into a powerful representation learning framework in computer vision and natural language processing. However, their application to graph data has resulted in limited performance due to the non-Euclidean and complex structure of graphs compared to images or text, as well as the limitations of conventional autoencoder architectures. In this paper, we investigate factors impacting the performance of autoencoders on graph data and propose a novel autoencoder model for graph representation learning. Our model incorporates a hierarchical adaptive masking mechanism to incrementally increase the difficulty of training to mimic the process of human cognitive learning and a trainable corruption scheme to enhance the robustness of learned representations. Through extensive experimentation on ten benchmark datasets, we demonstrate the superiority of our proposed method over state-of-the-art graph representation learning models.

Do the paper’s connections to existing work disclose its citation impact? A study based on graph representation learning

Influential scientific papers tend to be primarily based on combinations of prior works. However, assessing the potential impact of a new scientific paper remains a challenging task. In this article, we introduce an innovative framework to investigate the relationship between the embedding of citation networks and a paper’s future citation counts, based on the graph representation learning approach. First, we employ three Nobel Prize-winning topic papers from the Web of Science as our data source. Through data preprocessing and direct citation network modelling, we train the struc2vec model to obtain embeddings of papers’ citation network structure. Then, we perform visualisation and analysis on two types of networks. One is the direct-citation network, in which we identify four patterns of linkage between newly published papers and existing knowledge, and the other is the co-citation network, where we measure three structural variation indicators of new papers based on existing research findings. Finally, a statistical test is used to examine the predictive potentials of network embeddings. The results demonstrate that the structural features captured by the graph representation learning model can be used to predict a paper’s citation counts and impact. This article innovatively combines cluster analysis, visual analysis and statistical analysis to gain insights into the relationship between the hard-to-explain structural embeddings of newly published papers in a citation network and their future citations.

Heterogeneous network representation learning based on role feature extraction

Since most of the real-world networks are heterogeneous, existing methods cannot characterize the roles of nodes in heterogeneous networks. The neighborhood structure of nodes in heterogeneous networks largely determines the node roles, and the basic statistical features of nodes describe the topology of nodes to some extent, so extracting structural features from the adjacency matrix of networks is crucial for role-oriented network representation learning(structural equivalence). Therefore, in this paper, we propose a heterogeneous network representation learning model based on role feature extraction, called HRFE(Heterogeneous Network Representation Learning for Role Feature Extraction). Firstly, we perform feature extraction for each node in the heterogeneous network to obtain a high-dimensional feature matrix, then perform role discovery using non-negative matrix decomposition techniques to obtain a role-based node representation, and finally verify the effectiveness of the model HRFE through experiments on a large number of real datasets.

An interpretable graph representation learning model for accurate predictions of drugs aqueous solubility

As increasingly more data science-driven approaches have been applied for compound properties predictions in the domain of drug discovery, such kinds of methods have displayed considerable accuracy compared to conventional ones. In this work, we proposed an interpretable graph learning representation model, SolubNet, for drug aqueous solubility prediction. The comprehensive evaluation demonstrated that SolubNet can successfully capture the quantitative structure-property relationship and can be interpreted with layer-wise relevance propagation (LRP) algorithm regarding how prediction values are generated from original input structures. The key advantage of SolubNet lies in the fact that it includes 3 layers of Topology Adaptive Graph Convolutional Networks which can efficiently perceive chemical local environments. SolubNet showed high performance in several tasks for drugs’ aqueous solubility prediction. LRP revealed that SolubNet can identify high and low polar regions of a given molecule, assigning them reasonable weights to predict the final solubility, in a way highly compatible with chemists’ intuition. We are confident that such a flexible yet interpretable and accurate tool will largely enhance the efficiency of drug discovery, and will even contribute to the methodology development of computational pharmaceutics.

Molecular property prediction by semantic-invariant contrastive learning.

Contrastive learning has been widely used as pretext tasks for self-supervised pre-trained molecular representation learning models in AI-aided drug design and discovery. However, existing methods that generate molecular views by noise-adding operations for contrastive learning may face the semantic inconsistency problem, which leads to false positive pairs and consequently poor prediction performance. To address this problem, in this article, we first propose a semantic-invariant view generation method by properly breaking molecular graphs into fragment pairs. Then, we develop a Fragment-based Semantic-Invariant Contrastive Learning (FraSICL) model based on this view generation method for molecular property prediction. The FraSICL model consists of two branches to generate representations of views for contrastive learning, meanwhile a multi-view fusion and an auxiliary similarity loss are introduced to make better use of the information contained in different fragment-pair views. Extensive experiments on various benchmark datasets show that with the least number of pre-training samples, FraSICL can achieve state-of-the-art performance, compared with major existing counterpart models. The code is publicly available at https://github.com/ZiqiaoZhang/FraSICL.

Learning counterfactual outcomes of MOOC multiple learning behaviors

AbstractThe absence of counterfactual outcomes presents a fundamental challenge in causal inference. However, existing work typically does not apply to multiple learning behaviors of Massive Open Online Courses. This paper proposes a counterfactual representation learning model based on multitask learning, applicable to any dimension, and any type of treatment. The model consists of a potential outcome network and a propensity score encoder, which shares feature information from the base layer. The propensity scores calculated by the encoder are then utilized in the potential outcome network to mitigate selection bias. Experiments based on real‐world data sets demonstrate the superior performance of our model compared with baselines.

A representation learning model based on stochastic perturbation and homophily constraint

A representation learning model based on stochastic perturbation and homophily constraint

Edge-Based IIoT Malware Detection for Mobile Devices With Offloading

The advent of 5G brought new opportunities to leapfrog beyond current industrial Internet of Things (IoT). However, the ever growing IoT has also attracted adversaries to develop new malware attacks against various IoT applications. Although deep learning-based methods are expected to combat the sophisticated malwares by exploring the latent attack patterns, such detection can be hardly supported by battery-powered end devices, like Andriod-based smartphones. Edge computing enables near-real-time analysis of IoT data by migrating AI-enabled computation-intensive tasks from resource-constrained IoT devices to nearby edge servers, However, due to varying channel conditions and the demanding latency requirements of malware detection, it is challenging to coordinate the computing task offloading among multiple users. By leveraging the computation capacity and the proximity benefits of edge computing, we propose a hierarchical security framework for IoT malware detection. Considering the complexity of AI-enabled malware detection task, we provide a delay-aware computational offloading strategy with minimum delay. Specifically, we construct a coordinated representation learning model, named by Two Stream Attention-Caps (TSA-Caps) to capture the latent behavioral patterns of evolving malware attacks. Experimental results show that our system consistently outperforms the state-of-the-art systems in detection performance on four benchmark datasets.

Graph Representation Learning With Adaptive Metric

Contrastive learning has been widely used in graph representation learning, which extracts node or graph representations by contrasting positive and negative node pairs. It requires node representations (embeddings) to reflect their correlations in topology, increasing the similarities between an anchor node and its positive nodes, or reducing the similarities with its negative nodes in embedding space. However, most existing contrastive models measure similarities through a fixed metric that equally scores all sample pairs in a specific feature space, but ignores the varieties of node attributes and network topologies. Moreover, these fixed metrics are always defined explicitly and manually, which makes them unsuitable for applying to all graphs and networks. To solve these problems, we propose a novel graph representation learning model with an adaptive metric, called GRAM, which produces appropriate similarity scores of node pairs according to the different significance of each dimension in their embedding vectors and adaptive metrics based on data distribution. With these scores, it is better to train a graph encoder and obtain representative embeddings. Experimental results show that GRAM has strong competitiveness in multiple tasks.

BridgeTower: Building Bridges between Encoders in Vision-Language Representation Learning

Vision-Language (VL) models with the Two-Tower architecture have dominated visual-language representation learning in recent years. Current VL models either use lightweight uni-modal encoders and learn to extract, align and fuse both modalities simultaneously in a deep cross-modal encoder, or feed the last-layer uni-modal representations from the deep pre-trained uni-modal encoders into the top cross-modal encoder. Both approaches potentially restrict vision-language representation learning and limit model performance. In this paper, we propose BridgeTower, which introduces multiple bridge layers that build a connection between the top layers of uni-modal encoders and each layer of the cross-modal encoder. This enables effective bottom-up cross-modal alignment and fusion between visual and textual representations of different semantic levels of pre-trained uni-modal encoders in the cross-modal encoder. Pre-trained with only 4M images, BridgeTower achieves state-of-the-art performance on various downstream vision-language tasks. In particular, on the VQAv2 test-std set, BridgeTower achieves an accuracy of 78.73%, outperforming the previous state-of-the-art model METER by 1.09% with the same pre-training data and almost negligible additional parameters and computational costs. Notably, when further scaling the model, BridgeTower achieves an accuracy of 81.15%, surpassing models that are pre-trained on orders-of-magnitude larger datasets. Code and checkpoints are available at https://github.com/microsoft/BridgeTower.

Robust Causal Graph Representation Learning against Confounding Effects

The prevailing graph neural network models have achieved significant progress in graph representation learning. However, in this paper, we uncover an ever-overlooked phenomenon: the pre-trained graph representation learning model tested with full graphs underperforms the model tested with well-pruned graphs. This observation reveals that there exist confounders in graphs, which may interfere with the model learning semantic information, and current graph representation learning methods have not eliminated their influence. To tackle this issue, we propose Robust Causal Graph Representation Learning (RCGRL) to learn robust graph representations against confounding effects. RCGRL introduces an active approach to generate instrumental variables under unconditional moment restrictions, which empowers the graph representation learning model to eliminate confounders, thereby capturing discriminative information that is causally related to downstream predictions. We offer theorems and proofs to guarantee the theoretical effectiveness of the proposed approach. Empirically, we conduct extensive experiments on a synthetic dataset and multiple benchmark datasets. Experimental results demonstrate the effectiveness and generalization ability of RCGRL. Our codes are available at https://github.com/hang53/RCGRL.

Contrastive Pre-training with Adversarial Perturbations for Check-In Sequence Representation Learning

A core step of mining human mobility data is to learn accurate representations for user-generated check-in sequences. The learned representations should be able to fully describe the spatial-temporal mobility patterns of users and the high-level semantics of traveling. However, existing check-in sequence representation learning is usually implicitly achieved by end-to-end models designed for specific downstream tasks, resulting in unsatisfactory generalizable abilities and poor performance. Besides, although the sequence representation learning models that follow the contrastive learning pre-training paradigm have achieved breakthroughs in many fields like NLP, they fail to simultaneously consider the unique spatial-temporal characteristics of check-in sequences and need manual adjustments on the data augmentation strategies. So, directly applying them to check-in sequences cannot yield a meaningful pretext task. To this end, in this paper we propose a contrastive pre-training model with adversarial perturbations for check-in sequence representation learning (CACSR). Firstly, we design a novel spatial-temporal augmentation block for disturbing the spatial-temporal features of check-in sequences in the latent space to relieve the stress of designing manual data augmentation strategies. Secondly, to construct an effective contrastive pretext task, we generate “hard” positive and negative pairs for the check-in sequence by adversarial training. These two designs encourage the model to capture the high-level spatial-temporal patterns and semantics of check-in sequences while ignoring the noisy and unimportant details. We demonstrate the effectiveness and versatility of CACSR on two kinds of downstream tasks using three real-world datasets. The results show that our model outperforms both the state-of-the-art pre-training methods and the end-to-end models.

A Lightweight Multi-View Learning Approach for Phishing Attack Detection Using Transformer with Mixture of Experts

Phishing poses a significant threat to the financial and privacy security of internet users and often serves as the starting point for cyberattacks. Many machine-learning-based methods for detecting phishing websites rely on URL analysis, offering simplicity and efficiency. However, these approaches are not always effective due to the following reasons: (1) highly concealed phishing websites may employ tactics such as masquerading URL addresses to deceive machine learning models, and (2) phishing attackers frequently change their phishing website URLs to evade detection. In this study, we propose a robust, multi-view Transformer model with an expert-mixture mechanism for accurate phishing website detection utilizing website URLs, attributes, content, and behavioral information. Specifically, we first adapted a pretrained language model for URL representation learning by applying adversarial post-training learning in order to extract semantic information from URLs. Next, we captured the attribute, content, and behavioral features of the websites and encoded them as vectors, which, alongside the URL embeddings, constitute the website’s multi-view information. Subsequently, we introduced a mixture-of-experts mechanism into the Transformer network to learn knowledge from different views and adaptively fuse information from various views. The proposed method outperforms state-of-the-art approaches in evaluations of real phishing websites, demonstrating greater performance with less label dependency. Furthermore, we show the superior robustness and enhanced adaptability of the proposed method to unseen samples and data drift in more challenging experimental settings.

Vulnerability detection through cross-modal feature enhancement and fusion

Software vulnerability detection is critical to computer security. Most existing vulnerability detection methods use single modal-based vulnerability detection models, which cannot effectively extract cross-modal features. To solve this problem, we propose a new multimodal deep learning based vulnerability detection method through a cross-modal feature enhancement and fusion. Firstly, we utilize a special compilation and debugging method to obtain the alignment relationship between source code statements and assembly instructions, as well as between source code variables and assembly code registers. Based on this alignment relationship and program slicing technology, we propose a cross-slicing method to generate bimodal program slices. Then, we propose a cross-modal feature enhanced code representation learning model to capture the fine-grained semantic correlation between source code and assembly code by using the co-attention mechanisms. Finally, vulnerability detection is achieved by feature level fusion of semantic features captured in fine-grained aligned source code and assembly code. Extensive experiments show that our method improves the performance of vulnerability detection compared with state-of-the-art methods. Specifically, our method achieves an accuracy of 97.4% and an F1-measure of 93.4% on the SARD dataset. An average accuracy of 95.4% and an F1-measure of 89.1% on two real-world software projects (i.e., FFmpeg and OpenSSL) is also achieved by our method, improving over SOTA method 4.5% and 2.9%.

TDRLM: Stylometric learning for authorship verification by Topic-Debiasing

Authorship verification is the task of determining if a candidate is the real author of the provided sequences of texts. Stylometric learning is the study of the writing style by analyzing the textual data through computational approaches for many linguistic tasks such as authorship verification, authorship identification, etc. In recent years, Deep learning has been used as an effective solution to stylometric learning for authorship verification. However, most of the existing studies do not consider the semantic biases captured by the model, and the learned stylometric features include a strong indicator of the topical biases. During the extraction of stylometric features, we found that specific words related to the latent topics impact the performance of machine learning algorithms and, therefore, are not generalizable toward out-of-sample authors. In this paper, we propose a latent topic score dictionary with a Topic-Debiasing Representation Learning Model (TDRLM) for stylometric representation learning on the problem of authorship verification. The model applies position-specific topic scores on a topic-debiasing attention mechanism in order to adjust the tokenized texts based on the topical bias. The experimental results show that our proposed approach achieves the best performance with the highest Area Under Curve (AUC) of 92.47% for the Twitter-Foursquare dataset and 93.11% for the (ICWSM) Twitter dataset, both better than the state-of-the-art stylometric learning models and the up-trained language models.

A Hybrid Continuous-Time Dynamic Graph Representation Learning Model by Exploring Both Temporal and Repetitive Information

Recently, dynamic graph representation learning has attracted more and more attention from both academic and industrial communities due to its capabilities of capturing different real-world phenomena. For a dynamic graph represented as a sequence of timestamped events, there are two kinds of evolutionary essences: temporal and repetitive information. At present, the temporal information of interactions (e.g., timestamps) have been deeply explored. However, as another vital nature of dynamic graphs, the repetitive information of interactions between two nodes is neglected, which may lead to inaccurate node representation. To address this issue, we propose a novel continuous-time dynamic graph representation learning model, which consists of a node-level-memory based module, a historical high-order neighborhood based vertical aggregation module and a repetitive-topological information based horizontal aggregation module. In particular, to characterize the evolving pattern of the repetitive information of interactions between a pair of nodes, we put forward a repetitive-interaction based attention mechanism to integrate the two key attributes (i.e., the content and the number of interactions) of repetitive interactions at different moments, based on the insight that the repetitive behaviors of nodes are widespread and essential. We conduct extensive experiments including future link prediction tasks (for transductive and inductive learning) and dynamic node classification task, and results on three real-life dynamic graph datasets demonstrate that the proposed method significantly outperforms state-of-the-art baselines, for both observed nodes and new ones.

Micro-supervised Disturbance Learning: A Perspective of Representation Probability Distribution.

The instability is shown in the existing methods of representation learning based on Euclidean distance under a broad set of conditions. Furthermore, the scarcity and high cost of labels prompt us to explore more expressive representation learning methods which depends on as few labels as possible. To address above issues, the small-perturbation ideology is firstly introduced on the representation learning model based on the representation probability distribution. The positive small-perturbation information (SPI) which only depend on two labels of each cluster is used to stimulate the representation probability distribution and then two variant models are proposed to fine-tune the expected representation distribution of Restricted Boltzmann Machine (RBM), namely, Micro-supervised Disturbance Gaussian-binary RBM (Micro-DGRBM) and Micro-supervised Disturbance RBM (Micro-DRBM) models. The Kullback-Leibler (KL) divergence of SPI is minimized in the same cluster to promote the representation probability distributions to become more similar in Contrastive Divergence (CD) learning. In contrast, the KL divergence of SPI is maximized in the different clusters to enforce the representation probability distributions to become more dissimilar in CD learning. To explore the representation learning capability under the continuous stimulation of the SPI, we present a deep Microsupervised Disturbance Learning (Micro-DL) framework based on the Micro-DGRBM and Micro-DRBM models and compare it with a similar deep structure which has no external stimulation. Experimental results demonstrate that the proposed deep Micro-DL architecture shows better performance in comparison to the baseline method, the most related shallow models and deep frameworks for clustering.

Graph Learning for Interactive Threat Detection in Heterogeneous Smart Home Rule Data

The interactions among automation configuration rule data have led to undesired and insecure issues in smart homes, which are known as interactive threats. Most existing solutions use program analysis to identify interactive threats among automation rules, which is not suitable for closed-source platforms. Meanwhile, security policy-based solutions suffer from low detection accuracy because the pre-defined security policies in a single platform can hardly cover diverse interactive threat types across heterogeneous platforms. In this paper, we propose Glint, the first graph learning-based system for interactive threat detection in smart homes. We design a multi-scale graph representation learning model, called ITGNN, for both homogeneous and heterogeneous interaction graph pattern learning. To facilitate graph learning, we build large interaction graph training datasets by multi-domain data fusion from five different platforms. Moreover, Glint detects drifting samples with contrastive learning and improves the generalization ability with transfer learning across heterogeneous platforms. Our evaluation shows that Glint achieves 95.5% accuracy in detecting interactive threats across the five platforms. Besides, we examine a set of user-designed blueprints in the Home Assistant platform and reveal four new types of real-world interactive threats, called "action block", "action ablation", "trigger intake", and "condition duplicate", which are cross-platform interactive threats captured by Glint.