Representation Learning Model Research Articles

Cross-modal representation learning and generation

多媒体数据持续呈现爆发式增长并显现出异源异构的特性，因此跨模态学习领域研究逐渐引起学术和工业界的关注。跨模态表征与生成是跨模态学习的两大核心基础问题。跨模态表征旨在利用多种模态之间的互补性剔除模态之间的冗余，从而获得更为有效的特征表示；跨模态生成则是基于模态之间的语义一致性，实现不同模态数据形式上的相互转换，有助于提高不同模态间的迁移能力。本文系统地分析了国际与国内近年来跨模态表征与生成领域的重要研究进展，包括传统跨模态表征学习、多模态大模型表示学习、图像到文本的跨模态转换和跨模态图像生成。其中，传统跨模态表征学习探讨了跨模态统一表征和跨模态协同表征，多模态大模型表示学习探讨了基于Transformer的模型研究，图像到文本的跨模态转换探讨了图像视频的语义描述、视频字幕语义分析和视觉问答等领域的发展，跨模态图像生成从不同模态信息的跨模态联合表示方法、图像的跨模态生成技术和基于预训练的特定域图像生成阐述了跨模态生成方面的进展。本文详细综述了上述各个子领域研究的挑战性，对比了国内外研究方面的进展情况，梳理了发展脉络和学术研究的前沿动态。最后，根据上述分析展望了跨模态表征与生成的发展趋势和突破口。;Nowadays, with the booming of multimedia data, the character of multi-source and multi-modality of data has become a challenging problem in multimedia research. Its representation and generation can be as two key factors in cross-modal learning research. Cross-modal representation studies feature learning and information integration methods using multi-modal data. To get more effective feature representation, multimodality-between mutual benefits are required to be strengthened. Cross-modal generation is focused on the knowledge transfer mechanism across modalities. The modals-between semantic consistency can be used to realize data-interchangeable profiles of different modals. It is beneficial to improve modalities-between migrating ability. The literature review in cross-modal representation and generation are critically analyzed on the aspect of 1) traditional cross-modal representation learning, 2) big model for cross-modal representation learning, 3) image-to-text cross-modal conversion, joint representation, and 4) cross-modal image generation. Traditional cross-modal representation has two categories:joint representation and coordinated representation. Joint representation can yield multiple single-modal information to the joint representation space when each of single-modal information is processed through the coordinated representations, and cross-modal representations can be learnt mutually in terms of similarity constraints. Deep neural networks(DNNs) based self-supervised learning ability are activated to deal with largescale unlabeled data, especially for the Transformer-based methods. To enrich the supervised learning paradigm, the pretrained large models can yield large-scale unlabeled data to learn training, and a downstream tasks-derived small amount of labeled data is used for model fine-tuning. The pre-trained model has better versatility and transfering ability compared to the trained model for specific tasks, and the fine-tuned model can be used to optimize downstream tasks as well. The developmentof cross-modal synthesis(a.k.a. image caption or video caption) methods have been summarized, including end-toend, semantic-based, and stylize-based methods. In addition, current situation of cross-modal conversion between image and text has beenanalyzed, including image caption, video caption, and visual question answering. The cross-modal generation methods are summarized as well in relevance to the joint representation of cross-modal information, image generation, text-image cross-modal generation, and cross-modal generation based on pre-trained models. In recent years, generative adversarial networks(GANs) and denoising diffusion probabilistic models(DDPMs) have been faciliating in crossmodal generation tasks. Thanks to the strong adaptability and generation ability of DDPM models, cross-modal generation research can be developed and the constraints of vulnerable textures are optimized to a certain extent. The growth of GAN-based and DDPM-based methods are summarized and analyzed further.

Improving Link Prediction in Network Representation Learning with Feature Fusion and Local Outlier Factor

Complex networks are a diverse set of networks found in various fields, such as social, technological, and biological networks. One important task in complex network analysis is link prediction, which involves detecting missing links or predicting future link formation. Many methods based on network structure analysis have been developed for link prediction, including network representation learning (NRL) models that represent nodes in a low-dimensional space. Fusion-based attributed NRL methods are particularly effective, as they capture both content and structure information. However, NRL models for link prediction are binary classification models, which face challenges in identifying negative links and prioritizing predicted links. To address these challenges, we propose a novel approach that treats link prediction as a novelty detection problem. Our approach uses the Local Outlier Factor (LOF) algorithm to quantify the novelty of non-existent links based on the representations of existing links. Our experimental results show that our proposed approach outperforms existing methods, particularly when used with fusion-based attributed NRL models

Discriminative Local Representation Learning for Cross-Modality Visible-Thermal Person Re-Identification

Visible-thermal person re-identification (VTReID) is a rising and challenging cross-modality retrieval task in intelligent video surveillance systems. Most attention architectures cannot explore the discriminative person representations for VTReID, especially in the thermal modality. In addition, the fine-grained middle-level semantic information has received much less attention in the part-based approaches for the cross-modality pedestrian retrieval task, resulting in limited generalization capability and poor representation robustness. This paper proposes a simple yet powerful discriminative local representation learning (DLRL) model to capture the robust local fine-grained feature representations and explore the rich semantic relationship between the learned part features. Specifically, an efficient contextual attention aggregation module (CAAM) is designed to strengthen the discriminative capability of the feature representations and explore the contextual cues for visible and thermal modalities. Then, an integrated middle-high feature learning (IMHF) method is introduced to capture the part-level salient representations, which handles the ambiguous modality discrepancy in both discriminative middle-level and robust high-level information. Moreover, a part-guided graph convolution module (PGCM) is constructed to mine the structural relationship among the part representations within each modality. The quantitative and qualitative experiments on the two benchmark datasets demonstrate that the proposed DLRL model significantly outperforms state-of-the-art methods and achieves rank-1/mAP accuracy of 92.77%/82.05% on the RegDB dataset and 63.04%/60.58% on the SYSU-MM01 dataset.

Road-Type Classification with Deep AutoEncoder.

Machine learning algorithms are among the driving forces towards the success of intelligent road network systems design. Such algorithms allow for the design of systems that provide safe road usage, efficient infrastructure, and traffic flow management. One such application of machine learning in intelligent road networks is classifying different road network types that provide useful traffic information to road users. We propose a deep autoencoder model for representation learning to classify road network types. Each road segment node is represented as a feature vector. Unlike existing graph embedding methods that perform road segment embedding using the neighbouring road segments, the proposed method performs embedding directly on the road segment vectors. The proposed method performs embedding directly on the road segment vectors. Comparison with state-of-the-art graph embedding methods show that the proposed method outperforms graph convolution networks, GraphSAGE-MEAN, graph attention networks, and graph isomorphism network methods, and it achieves similar performance to GraphSAGE-MAXPOOL.

Future Event Prediction Based on Temporal Knowledge Graph Embedding

Accurate prediction of future events brings great benefits and reduces losses for society in many domains, such as civil unrest, pandemics, and crimes. Knowledge graph is a general language for describing and modeling complex systems. Different types of events continually occur, which are often related to historical and concurrent events. In this paper, we formalize the future event prediction as a temporal knowledge graph reasoning problem. Most existing studies either conduct reasoning on static knowledge graphs or assume knowledges graphs of all timestamps are available during the training process. As a result, they cannot effectively reason over temporal knowledge graphs and predict events happening in the future. To address this problem, some recent works learn to infer future events based on historical event-based temporal knowledge graphs. However, these methods do not comprehensively consider the latent patterns and influences behind historical events and concurrent events simultaneously. This paper proposes a new graph representation learning model, namely Recurrent Event Graph ATtention Network (RE-GAT), based on a novel historical and concurrent events attention-aware mechanism by modeling the event knowledge graph sequence recurrently. More specifically, our RE-GAT uses an attention-based historical events embedding module to encode past events, and employs an attention-based concurrent events embedding module to model the associations of events at the same timestamp. A translation-based decoder module and a learning objective are developed to optimize the embeddings of entities and relations. We evaluate our proposed method on four benchmark datasets. Extensive experimental results demonstrate the superiority of our RE-GAT model comparing to various baselines, which proves that our method can more accurately predict what events are going to happen.

Cyber-Attacks in IoT-enabled Cyber-physical Systems

Cyber physical systems (CPS) that are Internet of Things (IoT) enabled might be difficult to secure since security measures designed for general data / value through the development (IT / OT) systems may not work as well in a CPS environment. Consequently, this research provides a two-level ensemble attack detection and attribution framework created for CPS, and more particularly in an industrial control system (ICS). For identifying assaults in unbalanced ICS environments, a decision tree integrated to an unique ensemble deep representation learning model is created at the first extent. An ensemble deep neural network is created for assault features at the second level. Applying actual data collections from the gas pipeline and water treatment system, Findings show that the suggested type is more effective than other competing methods with a similar level of computational complexity.

Optimal Recommendation Models Based on Knowledge Representation Learning and Graph Attention Networks

Knowledge representation learning techniques process the knowledge graph, embedding entities and relationships into a continuous dense low-dimensional vector space, and providing the rich semantic association information between entities embedded in the knowledge graph to the recommendation module, improving recommendation performance and providing better interpretability. However, the knowledge representation learning techniques based on the translation model TransD with too many parameters and no association between entity representations are difficult to apply to large knowledge graph, as well as the problem is that most existing knowledge graph which based on recommendation systems ignore the different levels of importance that users attach to different relationships of items. To address these shortcomings, we propose an improved knowledge representation learning model Cluster TransD and a recommendation model Cluster TransD-GAT based on knowledge graph and graph attention networks, where the Cluster TransD model reduces the number of entity projections, makes the association between entity representations, reduces the computational pressure, and makes it better to be applied to the large knowledge graph, and the Cluster TransD-GAT model can capture the attention of different users to different relationships of items. Extensive comparison and ablation experiments on three real datasets show that the model proposed in this paper has significant performance improvements in terms of average ranking, accuracy and recall of the scoring function compared to other state-of-the-art models.

TransH-RA: A Learning Model of Knowledge Representation by Hyperplane Projection and Relational Attributes

The TransE model plays a key role in dealing with data sparsity and promotes the development of knowledge graphs completion. However, TransE has some difficulties in dealing with one-to-many, many-to-one, many-to-many and transmission relationships. In order to solve this problem, this paper proposes a knowledge representation learning model based on hyperplane projection and relational attributes, namely TransH-RA. First of all, we introduce the idea of hyperplane projection based on the TransE model, this idea is inspired by TransH, which makes different entities have different roles in a specific relationship, thus reducing the constraints of TransE translation rules, and map the head entity h and tail entity t to the plane of special relation r; Secondly, considering that it is not easy to identify different similar entities, the neighborhood information of entities is added to learn the neighborhood of entities around different entities; Then, in order to further strengthen the ability to deal with complex relationships, attribute features of relationships are added and attribute knowledge is embedded; Eventually, during the training of the model, the probability method is chosen to replace the head and tail entities. Link prediction experiments are conducted on the public datasets FB15K and WN18, and the triple classification experiments on the datasets WN11, FB13 and FB15K are carried out to analyze and verify the effectiveness of the proposed method. The evaluation results show that our method achieves the better results on MeanRank, Hits@10 and ACC indicators compared with TransE and TransH.

Providing Post-Hoc Explanation for Node Representation Learning Models Through Inductive Conformal Predictions

Learning with graph-structured data, such as social, biological, and financial networks, requires effective low-dimensional representations to handle their large and complex interactions. Recently, with the advances of neural networks and embedding algorithms, many unsupervised approaches have been proposed for many downstream tasks with promising results; however, there has been limited research on interpreting the unsupervised representations and, specifically, on understanding which parts of the neighboring nodes contribute to the representation of a node. To mitigate this problem, we propose a statistical framework to interpret the learned representations. Many of the existing works, which are designed for supervised node presentation models, compute the difference in prediction scores after perturbing the edges of a candidate explanation node; however, our proposed framework leverages a conformal prediction (CP)-based statistical test to verify the importance of the candidate node in each node representation. In our evaluation, our proposed framework was verified in many experimental settings and presented promising results compared to those of the recent baseline methods.

A Novel Deep Learning Representation for Industrial Control System Data

Feature extraction plays an important role in constructing artificial intelligence (AI) models of industrial control systems (ICSs). Three challenges in this field are learning effective representation from high-dimensional features, data heterogeneity, and data noise due to the diversity of data dimensions, formats and noise of sensors, controllers and actuators. Hence, a novel unsupervised learning autoencoder model is proposed for ICS data in this paper. Although traditional methods only capture the linear correlations of ICS features, our deep industrial representation learning model (DIRL) based on a convolutional neural network can mine high-order features, thus solving the problem of high-dimensional and heterogeneous ICS data. In addition, an unsupervised denoising autoencoder is introduced for noisy ICS data in DIRL. Training the denoising autoencoder allows the model to better mitigate the sensor noise problem. In this way, the representative features learned by DIRL could help to evaluate the safety state of ICSs more effectively. We tested our model with absolute and relative accuracy experiments on two large-scale ICS datasets. Compared with other popular methods, DIRL showed advantages in four common indicators of AI algorithms: accuracy, precision, recall, and F1-score. This study contributes to the effective analysis of large-scale ICS data, which promotes the stable operation of ICSs.

Design of a Visual Training System for Software Engineering Education Based on Knowledge Graphs

With the increase of the content and difficulty of software engineering education courses, software engineering education visual training system came into being, but the technology is not mature at present, and the representation learning algorithm part of the visual training system needs to be optimized. In order to solve this problem, the research proposes to optimize the take model by using the trans representation algorithm, and embed the optimized knowledge map into the new software engineering education visual training system. The performance of TEKEE model is verified by comparing TEKEE model with Trans E model, Trans D model and TEKED model. The experimental results show that the MR value of the optimized TEKEE model is 62, Hits@10 The value is 0.92, which is better than the other three representation learning models. In terms of the bearing capacity test of the visual training system, the response time of the business operation of the research and design training system is 1.22 seconds, and the CPU occupancy of the application server is 12.5%, all of which are normal. The experimental results show that the performance of the optimized TEKEE model has indeed been greatly improved, and the visual training system composed of the optimized knowledge map has a very good carrying capacity, which can provide a new idea for the software engineering education and training system.

GeoBERT: Pre-Training Geospatial Representation Learning on Point-of-Interest

Thanks to the development of geographic information technology, geospatial representation learning based on POIs (Point-of-Interest) has gained widespread attention in the past few years. POI is an important indicator to reflect urban socioeconomic activities, widely used to extract geospatial information. However, previous studies often focus on a specific area, such as a city or a district, and are designed only for particular tasks, such as land-use classification. On the other hand, large-scale pre-trained models (PTMs) have recently achieved impressive success and become a milestone in artificial intelligence (AI). Against this background, this study proposes the first large-scale pre-training geospatial representation learning model called GeoBERT. First, we collect about 17 million POIs in 30 cities across China to construct pre-training corpora, with 313 POI types as the tokens and the level-7 Geohash grids as the basic units. Second, we pre-train GeoEBRT to learn grid embedding in self-supervised learning by masking the POI type and then predicting. Third, under the paradigm of “pre-training + fine-tuning”, we design five practical downstream tasks. Experiments show that, with just one additional output layer fine-tuning, GeoBERT outperforms previous NLP methods (Word2vec, GloVe) used in geospatial representation learning by 9.21% on average in F1-score for classification tasks, such as store site recommendation and working/living area prediction. For regression tasks, such as POI number prediction, house price prediction, and passenger flow prediction, GeoBERT demonstrates greater performance improvements. The experiment results prove that pre-training on large-scale POI data can significantly improve the ability to extract geospatial information. In the discussion section, we provide a detailed analysis of what GeoBERT has learned from the perspective of attention mechanisms.

SimH: A Novel Representation Learning Model With Activation and Projection Mechanisms for COVID-19 Knowledge Bases.

The emergence of coronavirus disease 2019 (COVID-19) has had a significant impact on healthcare and the economy. With representation learning applied in constructing COVID-19 knowledge graphs, abundant COVID-19-related knowledge collected by clinicians and scientists all over the world can be utilized to deepen their understanding of the mechanism and related biological functions of the disease. However, most existing representation learning models cannot deal well with COVID-19 knowledge graph due to its low-connected star-like structure and various complex nonlinear relationships. Besides, lacking reliable negative triplets is also a difficult problem, yet to be adequately resolved. In this article, we propose a novel representation learning model called translation on hyperplanes with an activation operation and similar semantic sampling (SimH) for COVID-19 knowledge graphs. In our proposed SimH, an activation operation is designed to provide additional interaction features for low-in-degree entities. Then the hyperplane projection technique is introduced to the distance-based scoring function so that those complex nonlinear relationships can be modeled with lower complexity maintained in comparison with other nonlinear models. Moreover, a negative triplet sampling method that adaptively replaces entities with similar semantics is introduced to generate reliable negative triplets. To verify the effectiveness of SimH, extensive experiments are conducted on the COVID-19-Concepts dataset. The experimental results show that our SimH model achieves significant improvements in prediction and classification accuracy over existing knowledge representation learning models.

Multi-Attribute Discriminative Representation Learning for Prediction of Adverse Drug-Drug Interaction.

Adverse drug-drug interaction (ADDI) is a significant life-threatening issue, posing a leading cause of hospitalizations and deaths in healthcare systems. This paper proposes a unified Multi-Attribute Discriminative Representation Learning (MADRL) model for ADDI prediction. Unlike the existing works that equally treat features of each attribute without discrimination and do not consider the underlying relationship among drugs, we first develop a regularized optimization problem based on CUR matrix decomposition for joint representative drug and discriminative feature selection such that the selected drugs and features can well approximate the original feature spaces and the critical factors discriminative to ADDIs can be properly explored. Different from the existing models that ignore the consistent and unique properties among attributes, a Generative Adversarial Network (GAN) framework is then designed to capture the inter-attribute shared and intra-attribute specific representations of adverse drug pairs for exploiting their consensus and complementary information in ADDI prediction. Meanwhile, MADRL is compatible with any kind of attributes and capable of exploring their respective effects on ADDI prediction. An iterative algorithm based on the alternating direction method of multipliers is developed for optimization. Experiments on publicly available dataset demonstrate the effectiveness of MADRL when compared with eleven baselines and its six variants.

RLFDDA: a meta-path based graph representation learning model for drug–disease association prediction

BackgroundDrug repositioning is a very important task that provides critical information for exploring the potential efficacy of drugs. Yet developing computational models that can effectively predict drug–disease associations (DDAs) is still a challenging task. Previous studies suggest that the accuracy of DDA prediction can be improved by integrating different types of biological features. But how to conduct an effective integration remains a challenging problem for accurately discovering new indications for approved drugs.MethodsIn this paper, we propose a novel meta-path based graph representation learning model, namely RLFDDA, to predict potential DDAs on heterogeneous biological networks. RLFDDA first calculates drug–drug similarities and disease–disease similarities as the intrinsic biological features of drugs and diseases. A heterogeneous network is then constructed by integrating DDAs, disease–protein associations and drug–protein associations. With such a network, RLFDDA adopts a meta-path random walk model to learn the latent representations of drugs and diseases, which are concatenated to construct joint representations of drug–disease associations. As the last step, we employ the random forest classifier to predict potential DDAs with their joint representations.ResultsTo demonstrate the effectiveness of RLFDDA, we have conducted a series of experiments on two benchmark datasets by following a ten-fold cross-validation scheme. The results show that RLFDDA yields the best performance in terms of AUC and F1-score when compared with several state-of-the-art DDAs prediction models. We have also conducted a case study on two common diseases, i.e., paclitaxel and lung tumors, and found that 7 out of top-10 diseases and 8 out of top-10 drugs have already been validated for paclitaxel and lung tumors respectively with literature evidence. Hence, the promising performance of RLFDDA may provide a new perspective for novel DDAs discovery over heterogeneous networks.

LIMU-BERT

Deep learning greatly empowers Inertial Measurement Unit (IMU) sensors for a wide range of sensing applications. Most existing works require substantial amounts of wellcurated labeled data to train IMU-based sensing models, which incurs high annotation and training costs. Compared with labeled data, unlabeled IMU data are abundant and easily accessible. This article presents a novel representation learning model that can make use of unlabeled IMU data and extract generalized rather than task-specific features. With the representations learned via our model, task-specific models trained with limited labeled samples can achieve superior performances in typical IMU sensing applications, such as Human Activity Recognition (HAR).

Corresponding Intelligent Calculation of the Whole Process of Building Civil Engineering Structure Based on Deep Learning

This study proposes the first fully deep learning-based structural response intelligent computing framework for civil engineering. For the first time, from the data side to the model side, the structural information of the structure itself and any loading system is comprehensively considered, which can be applied to materials, components, and even structures, system and other multi-level mechanical response prediction problems. First, according to the characteristics of structural calculation scenarios, a unified data interface mode for structural static characteristics is formulated, which preserves the original structural information input and effectively reduces manual intervention. On this basis, an attention mechanism and a deep cross network are introduced, and a structural static feature representation learning model PADCN is proposed, which can take into account the memory and generalization of structural static features, and mine the coupling relationship of different structural information. Then, the PADCN model is integrated with the dynamic feature prediction model Mechformer and connected with the designed general data interface to form an end-to-end data-driven structural response intelligent computing framework. In order to verify the validity of the framework, numerical experiments were carried out with the steel plate shear wall structure as the carrier, in which a data augmentation algorithm suitable for the field of structural calculation was proposed to alleviate the problem of lack of structural engineering data. The results show that the deep learning model based on this framework successfully predicts the whole-process nonlinear response of specimens with different structures, the simulation accuracy is better than that of the fine finite element model, and the computational efficiency exceeds the traditional numerical method by more than 1000 times, achieving a qualitative improvement. It is proven that the intelligent computing framework has excellent accuracy and efficiency.

Modeling Trajectories Obtained from External Sensors for Location Prediction via NLP Approaches.

Representation learning seeks to extract useful and low-dimensional attributes from complex and high-dimensional data. Natural language processing (NLP) was used to investigate the representation learning models to extract words’ feature vectors using their sequential order in the text via word embeddings and language models that maintain their semantic meaning. Inspired by NLP, in this paper, we tackle the representation learning problem for trajectories, using NLP methods to encode external sensors positioned in the road network and generate the features’ space to predict the next vehicle movement. We evaluate the vector representations of on-road sensors and trajectories using extrinsic and intrinsic strategies. Our results have shown the potential of natural language models to describe the space of features on trajectory applications as the next location prediction.

An Explainable Artificial-Intelligence-Based CNN Model for Knowledge Extraction From the Social Internet of Things: Proposing a New Model

Rich material is buried in the entity’s textual description information, its hierarchical-type information, and the graph’s topological structure information in the knowledge graph. As a result, these data can be a useful supplement to triple information in terms of improving performance. To appropriately exploit these social Internet of Things (IoT) data, entity details are first encoded using artificial-intelligence (AI)-based convolutional neural networks (CNNs). The unit vector and unit description vector are then projected into a given relational space using the hierarchical-type information, thus restricting its semantic content. The graph attention approach is then used to fuse the graph’s topological structure information to calculate the influence of various neighboring points on the entity. To deal with the data-sparse problem, the multihop relationship information among entities is calculated at the same time. Finally, a decoder is used to collect global information among dimensions. Link prediction experiments show that the multisource information combined knowledge representation learning (XAI-CNN) model based on explainable AI (XAI) can effectively use multisource social IoT information beyond triples and that other techniques may be better than the baseline model.

Effective Collaborative Representation Learning for Multilabel Text Categorization.

With the booming of deep learning, massive attention has been paid to developing neural models for multilabel text categorization (MLTC). Most of the works concentrate on disclosing word-label relationship, while less attention is taken in exploiting global clues, particularly with the relationship of document-label. To address this limitation, we propose an effective collaborative representation learning (CRL) model in this article. CRL consists of a factorization component for generating shallow representations of documents and a neural component for deep text-encoding and classification. We have developed strategies for jointly training those two components, including an alternating-least-squares-based approach for factorizing the pointwise mutual information (PMI) matrix of label-document and multitask learning (MTL) strategy for the neural component. According to the experimental results on six data sets, CRL can explicitly take advantage of the relationship of document-label and achieve competitive classification performance in comparison with some state-of-the-art deep methods.